Jul 22 12:38:50.909257 ip-10-0-137-115 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 12:38:50.909266 ip-10-0-137-115 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 12:38:50.909273 ip-10-0-137-115 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 12:38:50.909491 ip-10-0-137-115 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 12:39:01.084707 ip-10-0-137-115 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 12:39:01.084723 ip-10-0-137-115 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 083e34cfa23e43659900debb39223f71 -- Jul 22 12:40:58.680409 ip-10-0-137-115 systemd[1]: Starting Kubernetes Kubelet... Jul 22 12:40:59.155071 ip-10-0-137-115 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:59.155071 ip-10-0-137-115 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 12:40:59.155071 ip-10-0-137-115 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:59.155071 ip-10-0-137-115 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 12:40:59.155071 ip-10-0-137-115 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:59.159409 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.159259 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 12:40:59.165828 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165795 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.165828 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165824 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.165828 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165829 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.165828 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165832 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165837 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165842 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165845 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165848 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165851 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165854 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165857 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165859 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165862 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165865 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165868 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165871 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165874 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165878 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165881 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165883 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165886 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165889 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165891 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.166003 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165895 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165899 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165903 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165907 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165910 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165913 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165915 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165918 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165920 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165923 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165926 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165929 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165931 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165934 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165937 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165939 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165942 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165945 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165947 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.166589 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165949 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165952 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165955 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165957 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165960 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165963 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165965 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165968 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165971 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165973 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165976 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165978 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165982 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165985 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165987 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165990 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165994 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165996 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.165999 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166002 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.167350 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166004 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166007 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166010 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166013 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166016 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166020 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166023 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166039 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166042 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166045 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166048 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166050 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166054 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166058 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166061 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166064 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166067 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166069 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166072 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166074 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.167997 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166077 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166080 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166082 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166087 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166089 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166092 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166095 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166097 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166100 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166103 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166106 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166109 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166111 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166114 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166117 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166120 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.168501 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.166123 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168706 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168724 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168727 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168730 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168734 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168739 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168742 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168745 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168748 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168751 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168754 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168757 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168760 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168762 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168765 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168768 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168771 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168774 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168776 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.168901 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168779 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168781 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168791 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168794 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168796 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168799 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168802 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168804 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168808 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168810 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168813 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168815 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168818 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168821 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168824 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168827 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168830 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168833 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168835 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168838 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.169461 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168840 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168843 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168846 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168850 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168854 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168856 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168859 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168861 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168864 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168867 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168869 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168873 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168876 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168878 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168881 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168884 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168886 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168889 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168891 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168894 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.169953 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168896 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168899 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168902 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168905 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168907 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168910 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168915 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168918 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168922 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168924 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168927 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168929 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168932 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168934 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168939 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168943 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168946 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168949 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168952 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.170451 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168956 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168958 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168961 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168964 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168966 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168970 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168972 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168975 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168978 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168980 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168983 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168985 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168988 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168990 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168993 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168995 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.168998 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169001 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169005 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169008 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.170935 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169011 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169478 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169485 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169488 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169492 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169494 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169497 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169500 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169502 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169505 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169508 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169510 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169513 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169516 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169518 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169521 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169524 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169527 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169530 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.171434 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169532 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169535 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169537 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169540 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169542 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169545 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169547 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169550 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169552 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169555 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169558 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169562 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169567 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169570 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169597 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169600 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169603 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169607 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169610 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.171931 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169613 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169615 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169618 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169620 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169623 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169626 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169630 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169633 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169636 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169638 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169641 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169643 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169648 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169650 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169654 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169656 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169659 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169663 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169666 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.172422 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169670 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169673 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169675 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169678 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169681 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169683 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169686 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169689 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169692 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169695 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169697 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169700 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169703 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169706 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169709 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169712 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169714 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169717 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169720 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169722 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.172910 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169725 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169727 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169730 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169733 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169735 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169738 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169741 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169744 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169746 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169749 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169751 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169754 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169756 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169758 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169761 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169764 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169766 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169769 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169773 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169775 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.173411 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169778 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169780 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.169783 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169869 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169881 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169891 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169897 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169903 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169907 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169912 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169916 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169919 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169922 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169926 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169929 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169934 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169936 2557 flags.go:64] FLAG: --cgroup-root="" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169939 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169942 2557 flags.go:64] FLAG: --client-ca-file="" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169946 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169949 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169953 2557 flags.go:64] FLAG: --cluster-domain="" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169956 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169960 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 12:40:59.173884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169963 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169966 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169976 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169980 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169983 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169986 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169990 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169993 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.169996 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170000 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170003 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170007 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170010 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170013 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170016 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170019 2557 flags.go:64] FLAG: --enable-server="true" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170022 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170042 2557 flags.go:64] FLAG: --event-burst="100" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170046 2557 flags.go:64] FLAG: --event-qps="50" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170049 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170052 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170055 2557 flags.go:64] FLAG: --eviction-hard="" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170059 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170062 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170065 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 12:40:59.174505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170068 2557 flags.go:64] FLAG: --eviction-soft="" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170071 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170074 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170077 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170080 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170083 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170086 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170089 2557 flags.go:64] FLAG: --feature-gates="" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170092 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170095 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170098 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170101 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170104 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170108 2557 flags.go:64] FLAG: --help="false" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170111 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170114 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170117 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170120 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170123 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170127 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170129 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170132 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170135 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 12:40:59.175120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170138 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170141 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170144 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170146 2557 flags.go:64] FLAG: --kube-reserved="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170149 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170152 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170155 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170158 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170160 2557 flags.go:64] FLAG: --lock-file="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170163 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170166 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170169 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170175 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170178 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170181 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170184 2557 flags.go:64] FLAG: --logging-format="text" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170187 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170191 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170193 2557 flags.go:64] FLAG: --manifest-url="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170196 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170201 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170204 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170209 2557 flags.go:64] FLAG: --max-pods="110" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170212 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170216 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 12:40:59.175722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170219 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170222 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170225 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170228 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170231 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170239 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170242 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170246 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170250 2557 flags.go:64] FLAG: --pod-cidr="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170253 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170260 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170263 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170266 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170269 2557 flags.go:64] FLAG: --port="10250" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170272 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170275 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07491e3b643aa4575" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170278 2557 flags.go:64] FLAG: --qos-reserved="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170281 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170285 2557 flags.go:64] FLAG: --register-node="true" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170288 2557 flags.go:64] FLAG: --register-with-taints="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170291 2557 flags.go:64] FLAG: --registry-burst="10" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170295 2557 flags.go:64] FLAG: --registry-qps="5" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170298 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170301 2557 flags.go:64] FLAG: --reserved-memory="" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170309 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 12:40:59.176354 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170312 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170315 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170318 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170321 2557 flags.go:64] FLAG: --runonce="false" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170324 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170327 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170330 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170333 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170336 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170339 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170342 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170346 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170348 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170351 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170354 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170357 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170360 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170363 2557 flags.go:64] FLAG: --system-cgroups="" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170366 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170371 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170374 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170376 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170381 2557 flags.go:64] FLAG: --tls-min-version="" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170383 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170386 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 12:40:59.177003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170389 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170392 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170395 2557 flags.go:64] FLAG: --v="2" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170400 2557 flags.go:64] FLAG: --version="false" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170412 2557 flags.go:64] FLAG: --vmodule="" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170421 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.170426 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170558 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170563 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170566 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170569 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170572 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170575 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170578 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170581 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170584 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170586 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170589 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170592 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170595 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170597 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.177648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170602 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170605 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170608 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170611 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170614 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170616 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170621 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170624 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170627 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170630 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170632 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170635 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170637 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170640 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170642 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170645 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170650 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170655 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170657 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170660 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.178224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170663 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170678 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170681 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170684 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170686 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170689 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170692 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170695 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170698 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170700 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170703 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170705 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170708 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170711 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170714 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170716 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170719 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170723 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170728 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.178733 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170732 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170735 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170738 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170741 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170744 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170747 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170750 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170754 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170756 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170759 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170764 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170767 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170770 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170772 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170775 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170779 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170781 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170784 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170786 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170789 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.179224 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170792 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170795 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170797 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170800 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170803 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170806 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170810 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170814 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170817 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170819 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170822 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170827 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170830 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170832 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170835 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170837 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170840 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170843 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170845 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170848 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.179720 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170851 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170853 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170857 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170861 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170863 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.170866 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.171489 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.178851 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.178874 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.178898 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.178904 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178967 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178973 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178978 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178981 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.180227 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178984 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178987 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178991 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178994 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.178997 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179000 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179002 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179005 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179008 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179012 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179015 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179019 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179024 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179042 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179045 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179048 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179050 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179053 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179056 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179060 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.180609 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179063 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179066 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179068 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179071 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179074 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179078 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179081 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179085 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179089 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179092 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179095 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179098 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179101 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179103 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179106 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179108 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179111 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179114 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179116 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179118 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.181132 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179121 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179124 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179126 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179129 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179132 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179135 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179138 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179141 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179143 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179146 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179148 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179151 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179154 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179157 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179160 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179162 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179166 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179169 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179172 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.181614 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179175 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179177 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179180 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179183 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179185 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179188 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179191 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179194 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179196 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179199 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179201 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179204 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179207 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179209 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179212 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179215 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179218 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179220 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179223 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179226 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.182121 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179228 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179233 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179236 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179240 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179244 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179247 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179250 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179253 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179256 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179260 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179262 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179265 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179269 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179272 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179275 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.182621 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179277 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.179284 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179408 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179414 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179418 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179423 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179426 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179430 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179433 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179437 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179440 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179443 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179447 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179450 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:59.183008 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179453 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179456 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179459 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179462 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179465 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179467 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179470 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179473 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179476 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179479 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179481 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179484 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179487 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179490 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179492 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179495 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179498 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179501 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179504 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179506 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:59.183407 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179509 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179511 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179514 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179516 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179519 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179521 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179524 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179526 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179529 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179532 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179535 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179537 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179540 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179543 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179545 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179548 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179551 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179553 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179556 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179559 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:59.183900 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179562 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179565 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179567 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179570 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179573 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179575 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179578 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179580 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179583 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179586 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179588 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179591 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179593 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179596 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179599 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179602 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179604 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179607 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179610 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:59.184405 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179612 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179615 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179618 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179620 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179623 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179626 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179629 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179631 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179634 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179637 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179639 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179643 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179647 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179650 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179653 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179656 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179659 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179661 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179664 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:59.184861 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179667 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179669 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179672 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179674 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179678 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179680 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179683 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179686 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:40:59.179688 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.179693 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.179848 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.182712 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.183870 2557 server.go:1019] "Starting client certificate rotation" Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.183977 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:59.185346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.184041 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:59.186046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.186013 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:59.187262 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.187247 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:59.198993 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.198959 2557 log.go:25] "Validated CRI v1 runtime API" Jul 22 12:40:59.201492 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.201469 2557 log.go:25] "Validated CRI v1 image API" Jul 22 12:40:59.202489 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.202458 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 12:40:59.205598 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.205570 2557 fs.go:135] Filesystem UUIDs: map[1ad76e5f-34bb-4b47-bafa-92f7e8e8919e:/dev/nvme0n1p4 20828a58-3a77-4788-856f-d00ba0fade75:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 22 12:40:59.205670 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.205596 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 12:40:59.211789 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.211626 2557 manager.go:216] Machine: {Timestamp:2026-07-22 12:40:59.210123642 +0000 UTC m=+0.379293054 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123327 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270970d09c349809af9e6c19b92d94 SystemUUID:ec270970-d09c-3498-09af-9e6c19b92d94 BootID:083e34cf-a23e-4365-9900-debb39223f71 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:43:75:6a:c2:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:43:75:6a:c2:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:8d:da:16:87:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 12:40:59.211789 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.211769 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 12:40:59.211983 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.211889 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 12:40:59.214531 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.214508 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:40:59.215272 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.215228 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 12:40:59.215425 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.215277 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 12:40:59.216403 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.216384 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 12:40:59.216403 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.216404 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 12:40:59.216480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.216419 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:59.217167 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.217146 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:59.218259 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.218242 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 12:40:59.221960 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.221937 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:59.222119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222109 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 12:40:59.222150 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222142 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 22 12:40:59.222190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222155 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 12:40:59.222190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222177 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 12:40:59.222190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222189 2557 kubelet.go:398] "Adding apiserver pod source" Jul 22 12:40:59.222283 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222198 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 12:40:59.222678 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222651 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:59.222720 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.222687 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:59.224884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.224861 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 12:40:59.226575 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.226558 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 12:40:59.226630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.226596 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 12:40:59.227693 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227680 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 12:40:59.227734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227697 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 12:40:59.227734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227704 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 12:40:59.227734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227710 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 12:40:59.227734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227719 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 12:40:59.227734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227726 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227738 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227746 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227753 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227758 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227768 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 12:40:59.227867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.227775 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 12:40:59.229094 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.229078 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 12:40:59.229094 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.229094 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 12:40:59.233073 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.233019 2557 server.go:1268] "Started kubelet" Jul 22 12:40:59.233212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.233184 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 12:40:59.233480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.233420 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 12:40:59.233564 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.233504 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 12:40:59.235085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.234847 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 12:40:59.235085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.234899 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 12:40:59.235085 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.234899 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 12:40:59.235085 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.235021 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 12:40:59.235240 ip-10-0-137-115 systemd[1]: Started Kubernetes Kubelet. Jul 22 12:40:59.240132 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.240104 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 22 12:40:59.241949 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.241887 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 12:40:59.243044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.243013 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 12:40:59.243044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.243014 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 12:40:59.243589 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.243270 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.243669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.243610 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 12:40:59.243669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.243637 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 12:40:59.243669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.243649 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 12:40:59.244128 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244109 2557 factory.go:55] Registering systemd factory Jul 22 12:40:59.244128 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244131 2557 factory.go:223] Registration of the systemd container factory successfully Jul 22 12:40:59.244302 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244291 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 22 12:40:59.244302 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244301 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 22 12:40:59.244559 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244541 2557 factory.go:153] Registering CRI-O factory Jul 22 12:40:59.244559 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244561 2557 factory.go:223] Registration of the crio container factory successfully Jul 22 12:40:59.244653 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244625 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 12:40:59.244700 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244661 2557 factory.go:103] Registering Raw factory Jul 22 12:40:59.244700 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.244678 2557 manager.go:1196] Started watching for new ooms in manager Jul 22 12:40:59.245441 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.244475 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-115.ec2.internal.18c49d1374acf387 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-115.ec2.internal,UID:ip-10-0-137-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-115.ec2.internal,},FirstTimestamp:2026-07-22 12:40:59.232990087 +0000 UTC m=+0.402159498,LastTimestamp:2026-07-22 12:40:59.232990087 +0000 UTC m=+0.402159498,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-115.ec2.internal,}" Jul 22 12:40:59.245441 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.245441 2557 manager.go:349] Starting recovery of all containers Jul 22 12:40:59.255463 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.255416 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 12:40:59.255624 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.255578 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 12:40:59.255624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.255601 2557 manager.go:354] Recovery completed Jul 22 12:40:59.261505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.261335 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.263930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.263887 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.264063 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.263946 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.264063 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.263962 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.264613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.264595 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 12:40:59.264613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.264611 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 12:40:59.264613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.264597 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.264753 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.264633 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:59.265668 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.265652 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.265731 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.265681 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.265731 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.265692 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.266812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.266793 2557 policy_none.go:49] "None policy: Start" Jul 22 12:40:59.266812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.266814 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 12:40:59.266974 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.266828 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 12:40:59.268960 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.268938 2557 policy_none.go:47] "Start" Jul 22 12:40:59.275121 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.275090 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xd4tw" Jul 22 12:40:59.276619 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.276536 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-115.ec2.internal.18c49d137685072a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-115.ec2.internal,UID:ip-10-0-137-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-115.ec2.internal,},FirstTimestamp:2026-07-22 12:40:59.263928106 +0000 UTC m=+0.433097543,LastTimestamp:2026-07-22 12:40:59.263928106 +0000 UTC m=+0.433097543,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-115.ec2.internal,}" Jul 22 12:40:59.282224 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.282128 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-115.ec2.internal.18c49d13768569d2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-115.ec2.internal,UID:ip-10-0-137-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-115.ec2.internal,},FirstTimestamp:2026-07-22 12:40:59.263953362 +0000 UTC m=+0.433122774,LastTimestamp:2026-07-22 12:40:59.263953362 +0000 UTC m=+0.433122774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-115.ec2.internal,}" Jul 22 12:40:59.286942 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.286909 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xd4tw" Jul 22 12:40:59.289492 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.289416 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-115.ec2.internal.18c49d1376859dad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-115.ec2.internal,UID:ip-10-0-137-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-115.ec2.internal,},FirstTimestamp:2026-07-22 12:40:59.263966637 +0000 UTC m=+0.433136051,LastTimestamp:2026-07-22 12:40:59.263966637 +0000 UTC m=+0.433136051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-115.ec2.internal,}" Jul 22 12:40:59.314981 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.314956 2557 manager.go:337] "Starting Device Plugin manager" Jul 22 12:40:59.315124 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.315017 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 12:40:59.315124 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315045 2557 server.go:87] "Starting device plugin registration server" Jul 22 12:40:59.315350 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315335 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 12:40:59.315470 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315349 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 12:40:59.315470 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315451 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 12:40:59.315567 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315547 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 12:40:59.315567 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.315556 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 12:40:59.316189 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.316165 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 12:40:59.316315 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.316226 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.396341 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.396297 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 12:40:59.398301 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.397663 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 12:40:59.398301 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.397701 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 12:40:59.398301 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.397728 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 12:40:59.398301 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.397770 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 12:40:59.401086 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.401061 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:59.416462 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.416380 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.417493 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.417472 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.417604 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.417507 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.417604 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.417523 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.417604 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.417547 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.430368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.430344 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.430456 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.430376 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-115.ec2.internal\": node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.448376 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.448345 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.498088 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.498018 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal"] Jul 22 12:40:59.498277 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.498133 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.499682 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.499664 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.499792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.499696 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.499792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.499706 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.501714 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.501693 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.501845 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.501823 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.501905 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.501871 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.503244 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503224 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.503345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503258 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.503345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503271 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.503345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503225 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.503345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503337 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.503481 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.503353 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.505457 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.505439 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.505526 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.505489 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:59.506758 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.506740 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:59.506868 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.506766 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:59.506868 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.506776 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:59.539981 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.539947 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-115.ec2.internal\" not found" node="ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.544831 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.544804 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-115.ec2.internal\" not found" node="ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.548893 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.548862 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.645187 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.645149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c25f5faf03a7c576edb1dfa593890c7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-115.ec2.internal\" (UID: \"6c25f5faf03a7c576edb1dfa593890c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.645187 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.645190 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.645384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.645216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.649285 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.649257 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.746286 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c25f5faf03a7c576edb1dfa593890c7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-115.ec2.internal\" (UID: \"6c25f5faf03a7c576edb1dfa593890c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.746286 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746273 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c25f5faf03a7c576edb1dfa593890c7-config\") pod \"kube-apiserver-proxy-ip-10-0-137-115.ec2.internal\" (UID: \"6c25f5faf03a7c576edb1dfa593890c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.746440 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.746440 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.746440 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746388 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.746440 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.746413 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53e6933adde9eb15b9389942da22ccc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal\" (UID: \"53e6933adde9eb15b9389942da22ccc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.750367 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.750336 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.842627 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.842581 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.847783 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:40:59.847277 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:40:59.850921 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.850890 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:40:59.951746 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:40:59.951689 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:41:00.052358 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:00.052252 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:41:00.152959 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:00.152902 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:41:00.184110 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.184080 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 12:41:00.184522 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.184285 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 12:41:00.253421 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:00.253385 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:41:00.291150 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.291108 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 12:36:01 +0000 UTC" deadline="2028-02-25 00:40:12.84994503 +0000 UTC" Jul 22 12:41:00.291150 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.291142 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13979h59m12.558806924s" Jul 22 12:41:00.353989 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:00.353896 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-115.ec2.internal\" not found" Jul 22 12:41:00.395084 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.395019 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:41:00.396280 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.396260 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:41:00.396445 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.396292 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:41:00.396445 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.396303 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:41:00.445301 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.445274 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:41:00.481334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.481303 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:41:00.543739 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.543700 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" Jul 22 12:41:00.555681 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.555650 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:41:00.557742 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.557717 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" Jul 22 12:41:00.567166 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:00.567133 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:41:02.701790 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:02.701757 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:41:02.707330 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:02.707295 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c25f5faf03a7c576edb1dfa593890c7.slice/crio-8fa9f505cc104b92ce84f4e07701532c097dbe45c2416641ca9fa5546eb10395 WatchSource:0}: Error finding container 8fa9f505cc104b92ce84f4e07701532c097dbe45c2416641ca9fa5546eb10395: Status 404 returned error can't find the container with id 8fa9f505cc104b92ce84f4e07701532c097dbe45c2416641ca9fa5546eb10395 Jul 22 12:41:02.708381 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:02.708356 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e6933adde9eb15b9389942da22ccc2.slice/crio-00be9aae783218558c00056a0c5ebf47fee478d8fcb55b0d7e2e8c6d2ca3f35c WatchSource:0}: Error finding container 00be9aae783218558c00056a0c5ebf47fee478d8fcb55b0d7e2e8c6d2ca3f35c: Status 404 returned error can't find the container with id 00be9aae783218558c00056a0c5ebf47fee478d8fcb55b0d7e2e8c6d2ca3f35c Jul 22 12:41:02.713264 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:02.713242 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:41:03.259507 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.259475 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:41:03.304744 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.304702 2557 apiserver.go:52] "Watching apiserver" Jul 22 12:41:03.313939 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.313902 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 12:41:03.314448 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.314415 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-ghswk","openshift-multus/multus-n4dhk","openshift-network-diagnostics/network-check-target-zmhb7","openshift-network-operator/iptables-alerter-sm98p","kube-system/konnectivity-agent-bqh6h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92","openshift-cluster-node-tuning-operator/tuned-mlmtn","openshift-image-registry/node-ca-qx62d","openshift-multus/network-metrics-daemon-f8p8r","openshift-ovn-kubernetes/ovnkube-node-srz7w","kube-system/global-pull-secret-syncer-jxnk6","kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal","openshift-dns/node-resolver-z7jhj"] Jul 22 12:41:03.317252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.317207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.320006 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.319973 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.320589 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.320331 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bkr29\"" Jul 22 12:41:03.320589 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.320372 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.320788 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.320599 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 12:41:03.320788 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.320623 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.323000 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.322972 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 12:41:03.323000 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.322988 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.323516 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.323294 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 12:41:03.323751 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.323726 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v8zsk\"" Jul 22 12:41:03.323859 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.323789 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.323859 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.323802 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.325531 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.325509 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 12:41:03.325798 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.325781 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 12:41:03.325922 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.325901 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 12:41:03.325991 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.325925 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 12:41:03.326080 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.326063 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.326144 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.326096 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.326144 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.326108 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 12:41:03.326244 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.326064 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6f6vp\"" Jul 22 12:41:03.326302 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.326271 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:03.326377 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.326350 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:03.327847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.327744 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.330215 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.330188 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.331412 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.330914 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.331412 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.330955 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 12:41:03.331412 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.331053 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.331412 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.331219 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gsbd\"" Jul 22 12:41:03.332569 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.332547 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbtqc\"" Jul 22 12:41:03.332906 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.332799 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 12:41:03.332906 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.332824 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 12:41:03.333187 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.333165 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335664 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-etc-kubernetes\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e036619a-09e6-44e1-994f-e43c8b973d90-iptables-alerter-script\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335736 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlg9p\"" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335749 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-socket-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335784 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335802 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335850 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-system-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335873 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-socket-dir-parent\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-daemon-config\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335928 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e036619a-09e6-44e1-994f-e43c8b973d90-host-slash\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.335956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335958 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-netd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336002 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.335960 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336004 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwxb\" (UniqueName: \"kubernetes.io/projected/cb509632-9119-408a-a190-0ade82ad5846-kube-api-access-pfwxb\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-cnibin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336178 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-cni-binary-copy\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-k8s-cni-cncf-io\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336256 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-conf-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-device-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336304 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-sys-fs\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336327 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-kubelet\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336361 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-slash\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336376 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-systemd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336405 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-os-release\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-netns\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-registration-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.336611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-systemd-units\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336564 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-ovn\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-kubelet\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336612 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-config\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336662 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-env-overrides\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-netns\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336789 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-node-log\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.336859 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-script-lib\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337237 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggpk\" (UniqueName: \"kubernetes.io/projected/15b66aac-1c1f-443a-bc17-d534fe41cedf-kube-api-access-qggpk\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpkc\" (UniqueName: \"kubernetes.io/projected/e036619a-09e6-44e1-994f-e43c8b973d90-kube-api-access-wrpkc\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-log-socket\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337312 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-bin\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb509632-9119-408a-a190-0ade82ad5846-ovn-node-metrics-cert\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.337363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337361 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-bin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-multus\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337444 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q26gc\" (UniqueName: \"kubernetes.io/projected/78672f29-a012-4dec-a053-326746f6c266-kube-api-access-q26gc\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-var-lib-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-etc-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337556 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-hostroot\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.338123 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.337579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-multus-certs\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.339026 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.338853 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5qgh\"" Jul 22 12:41:03.339403 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.339379 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.339492 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.339433 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.339549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.339388 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.340833 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.340771 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:41:03.342860 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.342824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.343000 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.342939 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:03.343108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.343090 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 12:41:03.343247 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.343226 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 12:41:03.343670 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.343646 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j962k\"" Jul 22 12:41:03.348182 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.347941 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.350532 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.350478 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.350694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.350542 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:03.350694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.350561 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:03.350694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.350542 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:03.350694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.350598 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cfklb\"" Jul 22 12:41:03.361958 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.361929 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zftw8" Jul 22 12:41:03.367746 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.367687 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zftw8" Jul 22 12:41:03.426311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.426268 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 12:41:03.437844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437808 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437855 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-netd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437882 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-cni-binary-copy\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437913 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437944 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-sys-fs\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.437984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-slash\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438005 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-netd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438030 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438021 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b69eb66-ef86-415a-be83-810999d5cd1d-hosts-file\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69eb66-ef86-415a-be83-810999d5cd1d-tmp-dir\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-ovn\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438079 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438105 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-kubelet\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cnibin\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438182 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-lib-modules\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr6t\" (UniqueName: \"kubernetes.io/projected/1b69eb66-ef86-415a-be83-810999d5cd1d-kube-api-access-mdr6t\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438233 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-system-cni-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438259 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-multus\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438283 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-kubelet-config\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438305 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-var-lib-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-etc-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.438437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-hostroot\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438463 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-multus-certs\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e036619a-09e6-44e1-994f-e43c8b973d90-iptables-alerter-script\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e036619a-09e6-44e1-994f-e43c8b973d90-host-slash\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-konnectivity-ca\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-socket-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438595 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-socket-dir-parent\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438693 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-daemon-config\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-multus-certs\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438755 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70499e3b-ae4c-41d1-9549-0139ecf082e3-serviceca\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-slash\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438830 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-var-lib-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438838 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-cni-binary-copy\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438839 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-multus\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-etc-openvswitch\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438886 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.439160 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438891 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-socket-dir-parent\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmq85\" (UniqueName: \"kubernetes.io/projected/70499e3b-ae4c-41d1-9549-0139ecf082e3-kube-api-access-lmq85\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438940 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-os-release\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438941 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-sys-fs\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438984 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-ovn\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439011 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxb\" (UniqueName: \"kubernetes.io/projected/cb509632-9119-408a-a190-0ade82ad5846-kube-api-access-pfwxb\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.438704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e036619a-09e6-44e1-994f-e43c8b973d90-host-slash\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439018 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439056 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-cnibin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-k8s-cni-cncf-io\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439106 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-conf-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-agent-certs\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439219 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-var-lib-kubelet\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439270 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-kubelet\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.439990 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439275 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-device-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-kubelet\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439334 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-k8s-cni-cncf-io\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-kubelet\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-device-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439377 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-systemd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439397 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-socket-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-daemon-config\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-multus-conf-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439396 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-cnibin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-os-release\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-run-systemd\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439449 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e036619a-09e6-44e1-994f-e43c8b973d90-iptables-alerter-script\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439499 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-hostroot\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-os-release\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439529 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-netns\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439561 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xbj\" (UniqueName: \"kubernetes.io/projected/bede90bf-2533-4511-a623-8e219f89fefc-kube-api-access-g2xbj\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.440954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439609 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-run-netns\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-conf\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-registration-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439741 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-systemd-units\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439791 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-registration-dir\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439809 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-etc-tuned\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-config\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-systemd-units\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439918 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-env-overrides\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.439987 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78672f29-a012-4dec-a053-326746f6c266-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440085 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-modprobe-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440111 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-host\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440141 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-tmp\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-netns\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-node-log\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.441847 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-script-lib\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggpk\" (UniqueName: \"kubernetes.io/projected/15b66aac-1c1f-443a-bc17-d534fe41cedf-kube-api-access-qggpk\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440264 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-node-log\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpkc\" (UniqueName: \"kubernetes.io/projected/e036619a-09e6-44e1-994f-e43c8b973d90-kube-api-access-wrpkc\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440298 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70499e3b-ae4c-41d1-9549-0139ecf082e3-host\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-run-netns\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440324 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-dbus\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440430 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-config\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-env-overrides\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggr8q\" (UniqueName: \"kubernetes.io/projected/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-kube-api-access-ggr8q\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440585 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-log-socket\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440612 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-bin\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb509632-9119-408a-a190-0ade82ad5846-ovn-node-metrics-cert\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-bin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440810 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-log-socket\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440836 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-cni-bin\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysconfig\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.442689 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440839 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-host-var-lib-cni-bin\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440860 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb509632-9119-408a-a190-0ade82ad5846-ovnkube-script-lib\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.440968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-kubernetes\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441013 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-systemd\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441062 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/02761994-6a3f-4805-b3df-9db0a95767a4-kube-api-access-cp4b5\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q26gc\" (UniqueName: \"kubernetes.io/projected/78672f29-a012-4dec-a053-326746f6c266-kube-api-access-q26gc\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-etc-kubernetes\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441187 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441247 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-run\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-sys\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441358 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-etc-kubernetes\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441363 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-system-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15b66aac-1c1f-443a-bc17-d534fe41cedf-system-cni-dir\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.443528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.441521 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb509632-9119-408a-a190-0ade82ad5846-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.445423 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.445384 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb509632-9119-408a-a190-0ade82ad5846-ovn-node-metrics-cert\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.446781 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.446747 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:03.446781 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.446779 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:03.446964 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.446793 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:03.446964 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.446878 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.946859302 +0000 UTC m=+3.034348461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:03.448260 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.448233 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwxb\" (UniqueName: \"kubernetes.io/projected/cb509632-9119-408a-a190-0ade82ad5846-kube-api-access-pfwxb\") pod \"ovnkube-node-srz7w\" (UID: \"cb509632-9119-408a-a190-0ade82ad5846\") " pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.448924 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.448868 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggpk\" (UniqueName: \"kubernetes.io/projected/15b66aac-1c1f-443a-bc17-d534fe41cedf-kube-api-access-qggpk\") pod \"multus-n4dhk\" (UID: \"15b66aac-1c1f-443a-bc17-d534fe41cedf\") " pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.449934 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.449908 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q26gc\" (UniqueName: \"kubernetes.io/projected/78672f29-a012-4dec-a053-326746f6c266-kube-api-access-q26gc\") pod \"aws-ebs-csi-driver-node-zqb92\" (UID: \"78672f29-a012-4dec-a053-326746f6c266\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.450243 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.450220 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpkc\" (UniqueName: \"kubernetes.io/projected/e036619a-09e6-44e1-994f-e43c8b973d90-kube-api-access-wrpkc\") pod \"iptables-alerter-sm98p\" (UID: \"e036619a-09e6-44e1-994f-e43c8b973d90\") " pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.485761 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.485673 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" event={"ID":"6c25f5faf03a7c576edb1dfa593890c7","Type":"ContainerStarted","Data":"8fa9f505cc104b92ce84f4e07701532c097dbe45c2416641ca9fa5546eb10395"} Jul 22 12:41:03.487416 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.487372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" event={"ID":"53e6933adde9eb15b9389942da22ccc2","Type":"ContainerStarted","Data":"00be9aae783218558c00056a0c5ebf47fee478d8fcb55b0d7e2e8c6d2ca3f35c"} Jul 22 12:41:03.542282 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542186 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b69eb66-ef86-415a-be83-810999d5cd1d-hosts-file\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.542282 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69eb66-ef86-415a-be83-810999d5cd1d-tmp-dir\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542324 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b69eb66-ef86-415a-be83-810999d5cd1d-hosts-file\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cnibin\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542385 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cnibin\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-lib-modules\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr6t\" (UniqueName: \"kubernetes.io/projected/1b69eb66-ef86-415a-be83-810999d5cd1d-kube-api-access-mdr6t\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-system-cni-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.542494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-kubelet-config\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-konnectivity-ca\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70499e3b-ae4c-41d1-9549-0139ecf082e3-serviceca\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-lib-modules\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69eb66-ef86-415a-be83-810999d5cd1d-tmp-dir\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-kubelet-config\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542705 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-system-cni-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542745 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmq85\" (UniqueName: \"kubernetes.io/projected/70499e3b-ae4c-41d1-9549-0139ecf082e3-kube-api-access-lmq85\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.542812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542809 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-os-release\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-agent-certs\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542876 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-os-release\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542894 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-var-lib-kubelet\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.542977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xbj\" (UniqueName: \"kubernetes.io/projected/bede90bf-2533-4511-a623-8e219f89fefc-kube-api-access-g2xbj\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-conf\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-etc-tuned\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-modprobe-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-host\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543115 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-tmp\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543144 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70499e3b-ae4c-41d1-9549-0139ecf082e3-host\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-dbus\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543168 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70499e3b-ae4c-41d1-9549-0139ecf082e3-serviceca\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggr8q\" (UniqueName: \"kubernetes.io/projected/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-kube-api-access-ggr8q\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.543229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysconfig\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543248 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-konnectivity-ca\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543261 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-kubernetes\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-systemd\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543320 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-kubernetes\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/02761994-6a3f-4805-b3df-9db0a95767a4-kube-api-access-cp4b5\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543369 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543389 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-modprobe-d\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543439 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-systemd\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-host\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.543472 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.543536 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:04.043517164 +0000 UTC m=+3.131006328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543395 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543571 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-run\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-sys\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543620 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.544025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543645 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543699 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-dbus\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543735 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-sys\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543729 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-run\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70499e3b-ae4c-41d1-9549-0139ecf082e3-host\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.543813 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:03.543871 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:04.043855715 +0000 UTC m=+3.131344897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543917 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysctl-conf\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-etc-sysconfig\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.543986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02761994-6a3f-4805-b3df-9db0a95767a4-var-lib-kubelet\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.544161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.544916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.544297 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.546241 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.546214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-tmp\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.546389 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.546245 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dd88ee1-0a1a-4206-b382-0785f8ba1fd0-agent-certs\") pod \"konnectivity-agent-bqh6h\" (UID: \"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0\") " pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.546481 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.546464 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02761994-6a3f-4805-b3df-9db0a95767a4-etc-tuned\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.552369 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.552305 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmq85\" (UniqueName: \"kubernetes.io/projected/70499e3b-ae4c-41d1-9549-0139ecf082e3-kube-api-access-lmq85\") pod \"node-ca-qx62d\" (UID: \"70499e3b-ae4c-41d1-9549-0139ecf082e3\") " pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.553536 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.553367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr6t\" (UniqueName: \"kubernetes.io/projected/1b69eb66-ef86-415a-be83-810999d5cd1d-kube-api-access-mdr6t\") pod \"node-resolver-z7jhj\" (UID: \"1b69eb66-ef86-415a-be83-810999d5cd1d\") " pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.555076 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.555045 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggr8q\" (UniqueName: \"kubernetes.io/projected/1a83f93a-e019-4b2c-8fcb-ba754ae39cba-kube-api-access-ggr8q\") pod \"multus-additional-cni-plugins-ghswk\" (UID: \"1a83f93a-e019-4b2c-8fcb-ba754ae39cba\") " pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.556919 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.556687 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4b5\" (UniqueName: \"kubernetes.io/projected/02761994-6a3f-4805-b3df-9db0a95767a4-kube-api-access-cp4b5\") pod \"tuned-mlmtn\" (UID: \"02761994-6a3f-4805-b3df-9db0a95767a4\") " pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:03.557085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.556990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xbj\" (UniqueName: \"kubernetes.io/projected/bede90bf-2533-4511-a623-8e219f89fefc-kube-api-access-g2xbj\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:03.633680 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.633391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" Jul 22 12:41:03.642776 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.642739 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:03.644295 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:03.644264 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78672f29_a012_4dec_a053_326746f6c266.slice/crio-33ae0f35a0ee974a0ce201ca69bf1ba131bbdc7ce2b02da1bd5795b2b240a985 WatchSource:0}: Error finding container 33ae0f35a0ee974a0ce201ca69bf1ba131bbdc7ce2b02da1bd5795b2b240a985: Status 404 returned error can't find the container with id 33ae0f35a0ee974a0ce201ca69bf1ba131bbdc7ce2b02da1bd5795b2b240a985 Jul 22 12:41:03.653146 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.653111 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4dhk" Jul 22 12:41:03.654348 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:03.654315 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb509632_9119_408a_a190_0ade82ad5846.slice/crio-0fc462196eccfbf56ceb02f186934e1a718da9f64ed647b82bbda4711383f8ad WatchSource:0}: Error finding container 0fc462196eccfbf56ceb02f186934e1a718da9f64ed647b82bbda4711383f8ad: Status 404 returned error can't find the container with id 0fc462196eccfbf56ceb02f186934e1a718da9f64ed647b82bbda4711383f8ad Jul 22 12:41:03.661685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.661651 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sm98p" Jul 22 12:41:03.664103 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:03.664062 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b66aac_1c1f_443a_bc17_d534fe41cedf.slice/crio-fb8a2bc9fb62623933bdd5858d234c79507360b67c15a42be6e09eda0c514701 WatchSource:0}: Error finding container fb8a2bc9fb62623933bdd5858d234c79507360b67c15a42be6e09eda0c514701: Status 404 returned error can't find the container with id fb8a2bc9fb62623933bdd5858d234c79507360b67c15a42be6e09eda0c514701 Jul 22 12:41:03.671035 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.671009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:03.672137 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:03.672106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode036619a_09e6_44e1_994f_e43c8b973d90.slice/crio-51c278da336061e9002b4c686edf8cf5819301059e251598045643a129025247 WatchSource:0}: Error finding container 51c278da336061e9002b4c686edf8cf5819301059e251598045643a129025247: Status 404 returned error can't find the container with id 51c278da336061e9002b4c686edf8cf5819301059e251598045643a129025247 Jul 22 12:41:03.678275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.678244 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qx62d" Jul 22 12:41:03.688051 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.688000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7jhj" Jul 22 12:41:03.697271 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.697230 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ghswk" Jul 22 12:41:03.705476 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:03.705435 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" Jul 22 12:41:04.047334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.047298 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.047366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.047412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047467 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047481 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047498 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:04.047503 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047503 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:04.047748 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047515 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:04.047748 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047549 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:05.04752817 +0000 UTC m=+4.135017344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:04.047748 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047568 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:05.047559054 +0000 UTC m=+4.135048218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:04.047748 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.047585 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:05.04757459 +0000 UTC m=+4.135063757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:04.370972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.370878 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:36:03 +0000 UTC" deadline="2027-12-29 19:47:13.011221731 +0000 UTC" Jul 22 12:41:04.370972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.370914 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12607h6m8.640312831s" Jul 22 12:41:04.480331 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.480296 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:04.480331 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.480325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:04.480594 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.480428 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:04.480594 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:04.480554 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:04.490780 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.490743 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"0fc462196eccfbf56ceb02f186934e1a718da9f64ed647b82bbda4711383f8ad"} Jul 22 12:41:04.492300 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.492227 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" event={"ID":"78672f29-a012-4dec-a053-326746f6c266","Type":"ContainerStarted","Data":"33ae0f35a0ee974a0ce201ca69bf1ba131bbdc7ce2b02da1bd5795b2b240a985"} Jul 22 12:41:04.493662 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.493591 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sm98p" event={"ID":"e036619a-09e6-44e1-994f-e43c8b973d90","Type":"ContainerStarted","Data":"51c278da336061e9002b4c686edf8cf5819301059e251598045643a129025247"} Jul 22 12:41:04.495393 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:04.495363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4dhk" event={"ID":"15b66aac-1c1f-443a-bc17-d534fe41cedf","Type":"ContainerStarted","Data":"fb8a2bc9fb62623933bdd5858d234c79507360b67c15a42be6e09eda0c514701"} Jul 22 12:41:04.849158 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:04.849123 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd88ee1_0a1a_4206_b382_0785f8ba1fd0.slice/crio-b7ccb7e6bdad3de3bf349368ad7cb910f77def6a659932476316e703cec87f1d WatchSource:0}: Error finding container b7ccb7e6bdad3de3bf349368ad7cb910f77def6a659932476316e703cec87f1d: Status 404 returned error can't find the container with id b7ccb7e6bdad3de3bf349368ad7cb910f77def6a659932476316e703cec87f1d Jul 22 12:41:04.850090 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:04.850058 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b69eb66_ef86_415a_be83_810999d5cd1d.slice/crio-2dcf49ab79f226abbe91a1130b38b31d44ca06b5d53c39fded89ca39112d77a2 WatchSource:0}: Error finding container 2dcf49ab79f226abbe91a1130b38b31d44ca06b5d53c39fded89ca39112d77a2: Status 404 returned error can't find the container with id 2dcf49ab79f226abbe91a1130b38b31d44ca06b5d53c39fded89ca39112d77a2 Jul 22 12:41:04.853618 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:04.853589 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02761994_6a3f_4805_b3df_9db0a95767a4.slice/crio-e459a7e6144e213127bce7d5070f8be10765033cefcee72ac3aaa1016e7d3585 WatchSource:0}: Error finding container e459a7e6144e213127bce7d5070f8be10765033cefcee72ac3aaa1016e7d3585: Status 404 returned error can't find the container with id e459a7e6144e213127bce7d5070f8be10765033cefcee72ac3aaa1016e7d3585 Jul 22 12:41:04.857026 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:04.856428 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70499e3b_ae4c_41d1_9549_0139ecf082e3.slice/crio-0a3be760aa9c8ee49c1745fee37c47289629604d91d05985cee4348fde5aca01 WatchSource:0}: Error finding container 0a3be760aa9c8ee49c1745fee37c47289629604d91d05985cee4348fde5aca01: Status 404 returned error can't find the container with id 0a3be760aa9c8ee49c1745fee37c47289629604d91d05985cee4348fde5aca01 Jul 22 12:41:04.857774 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:04.857751 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a83f93a_e019_4b2c_8fcb_ba754ae39cba.slice/crio-83ecb9a3a3c65bb67c6c4ec673a1b89cc2232c7bcc9f2240c40203bf57f0f7df WatchSource:0}: Error finding container 83ecb9a3a3c65bb67c6c4ec673a1b89cc2232c7bcc9f2240c40203bf57f0f7df: Status 404 returned error can't find the container with id 83ecb9a3a3c65bb67c6c4ec673a1b89cc2232c7bcc9f2240c40203bf57f0f7df Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.054183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054520 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054547 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054561 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054629 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:07.054607599 +0000 UTC m=+6.142096765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054651 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054725 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:07.05469239 +0000 UTC m=+6.142181552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.054522 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:05.054780 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.054774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:05.055230 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054883 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:05.055230 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.054923 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:07.054910732 +0000 UTC m=+6.142399891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:05.371525 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.371451 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:36:03 +0000 UTC" deadline="2028-05-05 22:14:21.07923166 +0000 UTC" Jul 22 12:41:05.371525 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.371520 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15681h33m15.707715807s" Jul 22 12:41:05.480384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.480289 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:05.480519 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:05.480446 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:05.502594 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.501692 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" event={"ID":"53e6933adde9eb15b9389942da22ccc2","Type":"ContainerStarted","Data":"8eee22d6bfdf1c8e5fc3a2dea26c5dfdd97028718ca02a37dcedc2212079e54c"} Jul 22 12:41:05.505035 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.504886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerStarted","Data":"83ecb9a3a3c65bb67c6c4ec673a1b89cc2232c7bcc9f2240c40203bf57f0f7df"} Jul 22 12:41:05.507447 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.507356 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qx62d" event={"ID":"70499e3b-ae4c-41d1-9549-0139ecf082e3","Type":"ContainerStarted","Data":"0a3be760aa9c8ee49c1745fee37c47289629604d91d05985cee4348fde5aca01"} Jul 22 12:41:05.508843 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.508809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" event={"ID":"02761994-6a3f-4805-b3df-9db0a95767a4","Type":"ContainerStarted","Data":"e459a7e6144e213127bce7d5070f8be10765033cefcee72ac3aaa1016e7d3585"} Jul 22 12:41:05.511383 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.511348 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7jhj" event={"ID":"1b69eb66-ef86-415a-be83-810999d5cd1d","Type":"ContainerStarted","Data":"2dcf49ab79f226abbe91a1130b38b31d44ca06b5d53c39fded89ca39112d77a2"} Jul 22 12:41:05.513619 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.513583 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bqh6h" event={"ID":"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0","Type":"ContainerStarted","Data":"b7ccb7e6bdad3de3bf349368ad7cb910f77def6a659932476316e703cec87f1d"} Jul 22 12:41:05.515540 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:05.515505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" event={"ID":"6c25f5faf03a7c576edb1dfa593890c7","Type":"ContainerStarted","Data":"546f30571503e4795284c17dd6ed667f86b208e8fd1866fcb430d414d9946f42"} Jul 22 12:41:06.480606 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:06.480538 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:06.481089 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:06.480681 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:06.481089 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:06.480773 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:06.481089 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:06.480856 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:06.524495 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:06.523231 2557 generic.go:356] "Generic (PLEG): container finished" podID="53e6933adde9eb15b9389942da22ccc2" containerID="8eee22d6bfdf1c8e5fc3a2dea26c5dfdd97028718ca02a37dcedc2212079e54c" exitCode=0 Jul 22 12:41:06.524495 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:06.524178 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" event={"ID":"53e6933adde9eb15b9389942da22ccc2","Type":"ContainerDied","Data":"8eee22d6bfdf1c8e5fc3a2dea26c5dfdd97028718ca02a37dcedc2212079e54c"} Jul 22 12:41:06.541737 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:06.540995 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-115.ec2.internal" podStartSLOduration=4.540972777 podStartE2EDuration="4.540972777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:05.532981737 +0000 UTC m=+4.620470920" watchObservedRunningTime="2026-07-22 12:41:06.540972777 +0000 UTC m=+5.628461961" Jul 22 12:41:07.074203 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:07.074160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:07.074395 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:07.074219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:07.074395 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:07.074276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:07.074520 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074415 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:07.074520 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074433 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:07.074520 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074445 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:07.074520 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074502 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.07448361 +0000 UTC m=+10.161972786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:07.074926 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074906 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:07.075016 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.074962 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.074944588 +0000 UTC m=+10.162433753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:07.075016 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.075014 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:07.075119 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.075057 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.075036426 +0000 UTC m=+10.162525591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:07.482191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:07.482108 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:07.482657 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:07.482244 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:08.479657 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:08.479620 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:08.479863 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:08.479773 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:08.480255 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:08.480071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:08.480255 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:08.480195 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:09.480674 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:09.480633 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:09.481238 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:09.480822 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:10.480118 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:10.479997 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:10.480293 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:10.480150 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:10.480583 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:10.480011 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:10.480583 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:10.480505 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:10.537931 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:10.537862 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" event={"ID":"53e6933adde9eb15b9389942da22ccc2","Type":"ContainerStarted","Data":"c604d574e43bb1f19ddfd3fc3207f3a8d3a391f7a52a1f8907452ad0301905f5"} Jul 22 12:41:10.557338 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:10.556122 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-115.ec2.internal" podStartSLOduration=8.5561005 podStartE2EDuration="8.5561005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:10.555965297 +0000 UTC m=+9.643454480" watchObservedRunningTime="2026-07-22 12:41:10.5561005 +0000 UTC m=+9.643589684" Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:11.114366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:11.114430 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:11.114460 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114521 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114542 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114555 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114560 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114611 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.114593581 +0000 UTC m=+18.202082754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114631 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114633 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.114622536 +0000 UTC m=+18.202111710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:11.114694 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.114667 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.114656701 +0000 UTC m=+18.202145861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:11.481484 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:11.480951 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:11.481484 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:11.481073 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:12.480393 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:12.480350 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:12.480889 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:12.480362 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:12.480889 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:12.480507 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:12.480889 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:12.480539 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:13.480688 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:13.480648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:13.481184 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:13.480810 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:14.479891 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:14.479857 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:14.479891 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:14.479886 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:14.480164 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:14.479988 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:14.480217 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:14.480151 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:15.480506 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:15.480456 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:15.481093 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:15.480619 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:16.480271 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:16.480227 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:16.480504 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:16.480284 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:16.480504 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:16.480386 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:16.480950 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:16.480534 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:17.479853 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:17.479813 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:17.480075 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:17.479955 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:18.479824 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:18.479793 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:18.480261 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:18.479892 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:18.480261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:18.479946 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:18.480261 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:18.480035 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:19.180555 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:19.180464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:19.180555 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:19.180515 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:19.180574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180634 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180657 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180686 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180703 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180731 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.180695504 +0000 UTC m=+34.268184712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180735 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180747 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.180740635 +0000 UTC m=+34.268229799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:19.180801 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.180779 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.180763875 +0000 UTC m=+34.268253037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:19.480647 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:19.480552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:19.481108 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:19.480676 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:20.480681 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:20.480640 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:20.480681 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:20.480673 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:20.481145 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:20.480801 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:20.481145 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:20.480917 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:21.481795 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:21.481756 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:21.482246 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:21.481990 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:22.480426 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:22.480390 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:22.480626 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:22.480390 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:22.480626 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:22.480513 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:22.480626 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:22.480612 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:23.480646 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.480450 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:23.481097 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:23.480753 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:23.566729 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.566687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" event={"ID":"02761994-6a3f-4805-b3df-9db0a95767a4","Type":"ContainerStarted","Data":"d0f85cdea216cde0c9b9d9094a0af2ef9de062d92bf20af04d17f04648de095a"} Jul 22 12:41:23.568297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.568269 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7jhj" event={"ID":"1b69eb66-ef86-415a-be83-810999d5cd1d","Type":"ContainerStarted","Data":"59440c05ebea48e812d90c83446d2a922fdd7ab2d3a95d4e7d9340f9b86df89d"} Jul 22 12:41:23.569701 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.569675 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bqh6h" event={"ID":"1dd88ee1-0a1a-4206-b382-0785f8ba1fd0","Type":"ContainerStarted","Data":"58a1c94bedc10a118a7ce5eb2682e150475d4617784356bf8546e403daffd94c"} Jul 22 12:41:23.571196 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.571168 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"0e2ccced6ad9c34b676cfe6ba6c5902cc61e50a7b4198372ed1834b346ac9b39"} Jul 22 12:41:23.572588 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.572568 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" event={"ID":"78672f29-a012-4dec-a053-326746f6c266","Type":"ContainerStarted","Data":"a35d410022de6b6b297cfb108d7b00620424155211c7c86c915194d45c137a8c"} Jul 22 12:41:23.573811 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.573791 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4dhk" event={"ID":"15b66aac-1c1f-443a-bc17-d534fe41cedf","Type":"ContainerStarted","Data":"e4f521e45d49559ef4f297c553d44006f2192da76f4d1c0c3e4a3d2a5cb15d18"} Jul 22 12:41:23.575368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.575344 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="b042add7d1d447619b695651d12e17eb3ac82047b6531deaf957fa5dba54f1b3" exitCode=0 Jul 22 12:41:23.575451 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.575409 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"b042add7d1d447619b695651d12e17eb3ac82047b6531deaf957fa5dba54f1b3"} Jul 22 12:41:23.576970 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.576948 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qx62d" event={"ID":"70499e3b-ae4c-41d1-9549-0139ecf082e3","Type":"ContainerStarted","Data":"9f2b13d257fceea4ce77ed9e93048f05f64268dddf5f54916469a600e1057a62"} Jul 22 12:41:23.586964 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.586919 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mlmtn" podStartSLOduration=4.506574142 podStartE2EDuration="22.586903688s" totalImagesPullingTime="18.080329546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.586583781 +0000 UTC m=+22.674072969" watchObservedRunningTime="2026-07-22 12:41:23.586903688 +0000 UTC m=+22.674392869" Jul 22 12:41:23.602264 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.602217 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bqh6h" podStartSLOduration=4.514966854 podStartE2EDuration="22.6022s" totalImagesPullingTime="18.087233146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.601696834 +0000 UTC m=+22.689186015" watchObservedRunningTime="2026-07-22 12:41:23.6022 +0000 UTC m=+22.689689181" Jul 22 12:41:23.648375 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.648319 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qx62d" podStartSLOduration=4.569978962 podStartE2EDuration="22.648294954s" totalImagesPullingTime="18.078315992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.647853209 +0000 UTC m=+22.735342389" watchObservedRunningTime="2026-07-22 12:41:23.648294954 +0000 UTC m=+22.735784134" Jul 22 12:41:23.664407 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.664356 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z7jhj" podStartSLOduration=4.5787204599999995 podStartE2EDuration="22.664340719s" totalImagesPullingTime="18.085620259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.664118779 +0000 UTC m=+22.751607959" watchObservedRunningTime="2026-07-22 12:41:23.664340719 +0000 UTC m=+22.751829900" Jul 22 12:41:23.689613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.689553 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n4dhk" podStartSLOduration=3.3703524 podStartE2EDuration="22.689535399s" totalImagesPullingTime="19.319182999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.688677155 +0000 UTC m=+22.776166337" watchObservedRunningTime="2026-07-22 12:41:23.689535399 +0000 UTC m=+22.777024581" Jul 22 12:41:23.754308 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.754011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:23.820423 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.820359 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:23.921834 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:23.921798 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:24.272102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.272059 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 12:41:24.408443 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.408326 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T12:41:24.272080922Z","UUID":"8c9a2874-94d9-43e5-8d11-2afe011d6c39","Handler":null,"Name":"","Endpoint":""} Jul 22 12:41:24.410135 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.410115 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 12:41:24.410273 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.410152 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 12:41:24.479813 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.479779 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:24.479813 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.479808 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:24.480101 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:24.479919 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:24.480101 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:24.480048 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:24.582439 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.582390 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sm98p" event={"ID":"e036619a-09e6-44e1-994f-e43c8b973d90","Type":"ContainerStarted","Data":"a74df5b931e29a1fb8d8a938153fa64aada3c0e8e6f3a0a5e844eb9762ebccb0"} Jul 22 12:41:24.587220 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.587173 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"c753bd59654cdd432286e0766ddfafe272ba2b6c306791b0e32b148736ef3e29"} Jul 22 12:41:24.587477 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.587263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"267609c6f511b86b2741702ede0d85866f1487c44aee809f28867c3ac41aff34"} Jul 22 12:41:24.587477 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.587314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"b97b7de7eaede240acd906575b7df6e184a1371561d76b26a1eee82da5251000"} Jul 22 12:41:24.587477 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.587362 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"350e78584e2c79200fab9a06f3745257cb71e409325abf5eb1bd27d3e87ad3b9"} Jul 22 12:41:24.587477 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.587407 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"04fa956e751fe5a1a9f5bceb8175670fa9e7ad1de03391e2cf689e4f25c3ee8d"} Jul 22 12:41:24.589444 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.589352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" event={"ID":"78672f29-a012-4dec-a053-326746f6c266","Type":"ContainerStarted","Data":"2bc638297bf6a63f9b4a557c49d8da5038916d54d30e41e7378d915a9964e098"} Jul 22 12:41:24.600142 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.600090 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sm98p" podStartSLOduration=16.399505083 podStartE2EDuration="23.600072978s" totalImagesPullingTime="7.200567895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:24.599597418 +0000 UTC m=+23.687086600" watchObservedRunningTime="2026-07-22 12:41:24.600072978 +0000 UTC m=+23.687562160" Jul 22 12:41:24.722388 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.722277 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:24.822974 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.822929 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:24.998380 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:24.998298 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:25.141690 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:25.141643 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:25.245346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:25.245302 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:25.462618 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:25.462526 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:25.480664 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:25.480631 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:25.480848 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:25.480803 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:25.563865 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:25.563829 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:26.273046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.273011 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:26.349821 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.349771 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:26.480593 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.480556 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:26.480593 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.480600 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:26.480856 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:26.480696 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:26.480856 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:26.480792 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:26.596482 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.596397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" event={"ID":"78672f29-a012-4dec-a053-326746f6c266","Type":"ContainerStarted","Data":"6211d1ed4c122900012c8bd3375aacd7d7f3e4cac3f17ae9d7a36a420b83a14f"} Jul 22 12:41:26.619017 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:26.618964 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqb92" podStartSLOduration=17.588344867 podStartE2EDuration="25.618944423s" totalImagesPullingTime="8.030599556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:26.618380904 +0000 UTC m=+25.705870087" watchObservedRunningTime="2026-07-22 12:41:26.618944423 +0000 UTC m=+25.706433604" Jul 22 12:41:27.181920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:27.181883 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:27.247136 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:27.247097 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:27.480738 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:27.480641 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:27.481199 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:27.480802 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:28.428792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.428509 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:28.429099 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.429077 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:28.480052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.480015 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:28.480216 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.480016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:28.480216 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:28.480116 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:28.480216 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:28.480200 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:28.604442 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.604409 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"9c2a3639fc88c831e6bffbaa2fd6ee7fc3ba105029b9e774a7578a573612cdc9"} Jul 22 12:41:28.606109 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.606082 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="cdfc0ae5a840aef71497c98441dd97cc37f50c46e3536366ecad92543cd9f1ef" exitCode=0 Jul 22 12:41:28.606214 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.606162 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"cdfc0ae5a840aef71497c98441dd97cc37f50c46e3536366ecad92543cd9f1ef"} Jul 22 12:41:28.606745 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.606725 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bqh6h" Jul 22 12:41:28.726195 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:28.726118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:29.185641 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:29.185547 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:29.247842 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:29.247810 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:29.480439 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:29.480358 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:29.480608 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:29.480475 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:30.480119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.479838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:30.480119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.479841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:30.480119 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:30.480110 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:30.480831 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:30.480217 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:30.612553 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.612510 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="d4a05b94d631cdd97f4ba31f88dbc17cd32059dabccb50e882440b3b57f6f184" exitCode=0 Jul 22 12:41:30.612726 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.612587 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"d4a05b94d631cdd97f4ba31f88dbc17cd32059dabccb50e882440b3b57f6f184"} Jul 22 12:41:30.616417 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.616368 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" event={"ID":"cb509632-9119-408a-a190-0ade82ad5846","Type":"ContainerStarted","Data":"182d3c0c807c6d6e82e251f85c85d057907411136464d2dd64f9252f7f966256"} Jul 22 12:41:30.616641 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.616622 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:30.616732 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.616699 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:30.633695 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.633658 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:30.673119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:30.673066 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" podStartSLOduration=9.881053079 podStartE2EDuration="29.673049406s" totalImagesPullingTime="19.791996327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:30.671509143 +0000 UTC m=+29.758998325" watchObservedRunningTime="2026-07-22 12:41:30.673049406 +0000 UTC m=+29.760538586" Jul 22 12:41:31.307594 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:31.307554 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:31.376950 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:31.376904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:31.481585 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:31.481543 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:31.481955 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:31.481668 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:31.618768 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:31.618661 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:31.633856 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:31.633828 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:41:32.268736 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.268680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f8p8r"] Jul 22 12:41:32.268946 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.268878 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:32.269010 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:32.268988 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:32.274426 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.274383 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxnk6"] Jul 22 12:41:32.274617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.274534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:32.274726 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:32.274677 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:32.275106 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.275072 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmhb7"] Jul 22 12:41:32.275264 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.275214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:32.275358 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:32.275334 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:32.623600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.623513 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="2cd446f6b257bfdf9b1382b03df4a6a6c6000f16408e7732218149a802eecba5" exitCode=0 Jul 22 12:41:32.623962 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:32.623595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"2cd446f6b257bfdf9b1382b03df4a6a6c6000f16408e7732218149a802eecba5"} Jul 22 12:41:33.210235 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.210204 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:33.302872 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.302833 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:33.479900 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.479824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:33.480069 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.479824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:33.480069 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:33.479963 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:33.480069 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:33.480057 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:33.747541 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.747457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:33.836616 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:33.836576 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:34.244833 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:34.244556 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:34.323193 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:34.323155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:34.480233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:34.480200 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:34.480418 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:34.480319 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxnk6" podUID="bf7e899a-13ae-4fa5-a1be-4677dbf43f4a" Jul 22 12:41:34.712813 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:34.712699 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:34.815451 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:34.815412 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:35.003139 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.003034 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:35.102603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.102565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:35.201679 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.201644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:35.201882 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.201705 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:35.201882 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.201761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:35.201882 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201833 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:35.201882 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201859 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:35.201882 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201832 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:35.202132 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201865 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:35.202132 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201926 2557 projected.go:196] Error preparing data for projected volume kube-api-access-kpxdz for pod openshift-network-diagnostics/network-check-target-zmhb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:35.202132 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201913 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret podName:bf7e899a-13ae-4fa5-a1be-4677dbf43f4a nodeName:}" failed. No retries permitted until 2026-07-22 12:42:07.201900835 +0000 UTC m=+66.289389994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret") pod "global-pull-secret-syncer-jxnk6" (UID: "bf7e899a-13ae-4fa5-a1be-4677dbf43f4a") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:35.202132 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201977 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs podName:bede90bf-2533-4511-a623-8e219f89fefc nodeName:}" failed. No retries permitted until 2026-07-22 12:42:07.201959822 +0000 UTC m=+66.289448987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs") pod "network-metrics-daemon-f8p8r" (UID: "bede90bf-2533-4511-a623-8e219f89fefc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:35.202132 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.201994 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz podName:56f83998-325d-4173-be07-bfce1b02e5c4 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:07.201983548 +0000 UTC m=+66.289472710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpxdz" (UniqueName: "kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz") pod "network-check-target-zmhb7" (UID: "56f83998-325d-4173-be07-bfce1b02e5c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:35.203987 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.203948 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:35.464764 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.464670 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z7jhj_1b69eb66-ef86-415a-be83-810999d5cd1d/dns-node-resolver/0.log" Jul 22 12:41:35.479931 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.479897 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:35.480157 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:35.479939 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:35.480157 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.480060 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f8p8r" podUID="bede90bf-2533-4511-a623-8e219f89fefc" Jul 22 12:41:35.480279 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:35.480236 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zmhb7" podUID="56f83998-325d-4173-be07-bfce1b02e5c4" Jul 22 12:41:36.070541 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.070507 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-115.ec2.internal" event="NodeReady" Jul 22 12:41:36.071017 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.070671 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 12:41:36.128762 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.128704 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gk4nb"] Jul 22 12:41:36.134114 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.134086 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jglnd"] Jul 22 12:41:36.134261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.134215 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.136898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.136875 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 12:41:36.136898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.136880 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 12:41:36.137084 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.136966 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kbrpx\"" Jul 22 12:41:36.137584 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.137559 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.141991 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.141826 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 12:41:36.142226 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.142193 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 12:41:36.142700 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.142666 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 12:41:36.147608 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.147346 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gqts7\"" Jul 22 12:41:36.148453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.148360 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jglnd"] Jul 22 12:41:36.149045 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.149020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gk4nb"] Jul 22 12:41:36.157058 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.157029 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cx4xd"] Jul 22 12:41:36.160679 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.160656 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.163787 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.163749 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 12:41:36.163935 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.163913 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvgjr\"" Jul 22 12:41:36.164213 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.164192 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 12:41:36.164292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.164276 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 12:41:36.165349 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.165288 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 12:41:36.171598 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.171576 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cx4xd"] Jul 22 12:41:36.210726 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.210670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca97e449-ed09-4e0b-81af-871d3951b7ca-tmp-dir\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.210926 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.210736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e449-ed09-4e0b-81af-871d3951b7ca-config-volume\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.210926 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.210796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.210926 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.210881 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9mt\" (UniqueName: \"kubernetes.io/projected/ca97e449-ed09-4e0b-81af-871d3951b7ca-kube-api-access-qn9mt\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.265481 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.265452 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qx62d_70499e3b-ae4c-41d1-9549-0139ecf082e3/node-ca/0.log" Jul 22 12:41:36.287492 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.287461 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:36.311594 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311555 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca97e449-ed09-4e0b-81af-871d3951b7ca-tmp-dir\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311601 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e449-ed09-4e0b-81af-871d3951b7ca-config-volume\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311634 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a71bf077-f306-4681-8f97-614a73c15e41-cert\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxp9\" (UniqueName: \"kubernetes.io/projected/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-api-access-znxp9\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311729 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6js\" (UniqueName: \"kubernetes.io/projected/a71bf077-f306-4681-8f97-614a73c15e41-kube-api-access-rt6js\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.311792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311753 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.311797 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.311876 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls podName:ca97e449-ed09-4e0b-81af-871d3951b7ca nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.811854421 +0000 UTC m=+35.899343595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls") pod "dns-default-gk4nb" (UID: "ca97e449-ed09-4e0b-81af-871d3951b7ca") : secret "dns-default-metrics-tls" not found Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311931 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9mt\" (UniqueName: \"kubernetes.io/projected/ca97e449-ed09-4e0b-81af-871d3951b7ca-kube-api-access-qn9mt\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.311987 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-crio-socket\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.312014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-data-volume\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.312127 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.312020 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca97e449-ed09-4e0b-81af-871d3951b7ca-tmp-dir\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.312406 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.312260 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca97e449-ed09-4e0b-81af-871d3951b7ca-config-volume\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.325312 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.325231 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9mt\" (UniqueName: \"kubernetes.io/projected/ca97e449-ed09-4e0b-81af-871d3951b7ca-kube-api-access-qn9mt\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.388231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.388194 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:36.413149 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a71bf077-f306-4681-8f97-614a73c15e41-cert\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.413149 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413141 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxp9\" (UniqueName: \"kubernetes.io/projected/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-api-access-znxp9\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413376 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6js\" (UniqueName: \"kubernetes.io/projected/a71bf077-f306-4681-8f97-614a73c15e41-kube-api-access-rt6js\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.413376 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413199 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413376 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.413292 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:36.413376 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413335 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413376 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.413363 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.913348697 +0000 UTC m=+36.000837896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : configmap "kube-rbac-proxy" not found Jul 22 12:41:36.413627 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-crio-socket\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413627 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-data-volume\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413627 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.413421 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:36.413627 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.413562 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.913526115 +0000 UTC m=+36.001015290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:36.413821 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413638 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-crio-socket\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.413821 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.413689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-data-volume\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.416660 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.416630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a71bf077-f306-4681-8f97-614a73c15e41-cert\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.422954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.422923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxp9\" (UniqueName: \"kubernetes.io/projected/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-api-access-znxp9\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.423107 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.422974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6js\" (UniqueName: \"kubernetes.io/projected/a71bf077-f306-4681-8f97-614a73c15e41-kube-api-access-rt6js\") pod \"ingress-canary-jglnd\" (UID: \"a71bf077-f306-4681-8f97-614a73c15e41\") " pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.458079 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.458038 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jglnd" Jul 22 12:41:36.480384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.480343 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:41:36.483310 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.483284 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:41:36.571221 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.571183 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qx5hf"] Jul 22 12:41:36.574897 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.574871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.577931 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.577859 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 12:41:36.577931 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.577883 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 12:41:36.578097 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.578009 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 12:41:36.578097 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.578009 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 12:41:36.578170 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.578118 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-js6dr\"" Jul 22 12:41:36.584669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.584599 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qx5hf"] Jul 22 12:41:36.612424 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.612379 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:36.716504 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.716466 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-cabundle\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.716693 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.716631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-key\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.716782 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.716680 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgqn\" (UniqueName: \"kubernetes.io/projected/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-kube-api-access-6mgqn\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.817801 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.817754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-cabundle\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.817972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.817874 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-key\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.817972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.817915 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgqn\" (UniqueName: \"kubernetes.io/projected/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-kube-api-access-6mgqn\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.817972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.817951 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:36.818111 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.818063 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:36.818161 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.818135 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls podName:ca97e449-ed09-4e0b-81af-871d3951b7ca nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.818117311 +0000 UTC m=+36.905606472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls") pod "dns-default-gk4nb" (UID: "ca97e449-ed09-4e0b-81af-871d3951b7ca") : secret "dns-default-metrics-tls" not found Jul 22 12:41:36.818603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.818579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-cabundle\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.820783 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.820760 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-signing-key\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.827601 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.827573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgqn\" (UniqueName: \"kubernetes.io/projected/adc13ec9-77dd-4be6-b0e5-fd3cd9671340-kube-api-access-6mgqn\") pod \"service-ca-58598695f7-qx5hf\" (UID: \"adc13ec9-77dd-4be6-b0e5-fd3cd9671340\") " pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.885484 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.885385 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qx5hf" Jul 22 12:41:36.918919 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.918883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.919078 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:36.918976 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:36.919078 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.919055 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:36.919078 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.919058 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:36.919211 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.919109 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.919095386 +0000 UTC m=+37.006584545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : configmap "kube-rbac-proxy" not found Jul 22 12:41:36.919211 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:36.919123 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.919115618 +0000 UTC m=+37.006604777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.084398 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.084372 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qx5hf"] Jul 22 12:41:37.086451 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.086421 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jglnd"] Jul 22 12:41:37.088524 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:37.088488 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc13ec9_77dd_4be6_b0e5_fd3cd9671340.slice/crio-6926c02e2c653b718f5fd71e5c6d3280b5921ede94a38eaffcdd23e582a00bcd WatchSource:0}: Error finding container 6926c02e2c653b718f5fd71e5c6d3280b5921ede94a38eaffcdd23e582a00bcd: Status 404 returned error can't find the container with id 6926c02e2c653b718f5fd71e5c6d3280b5921ede94a38eaffcdd23e582a00bcd Jul 22 12:41:37.089853 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:37.089672 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71bf077_f306_4681_8f97_614a73c15e41.slice/crio-441ae8b5aa1df92f6965e51ffc8b36b734201b0021d297b9a148cd9f3e586e2c WatchSource:0}: Error finding container 441ae8b5aa1df92f6965e51ffc8b36b734201b0021d297b9a148cd9f3e586e2c: Status 404 returned error can't find the container with id 441ae8b5aa1df92f6965e51ffc8b36b734201b0021d297b9a148cd9f3e586e2c Jul 22 12:41:37.479938 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.479842 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:41:37.480108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.480061 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:41:37.484524 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.484494 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:41:37.484687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.484505 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 12:41:37.485048 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.485025 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:41:37.485177 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.485070 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:41:37.485177 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.485114 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 12:41:37.532502 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.532467 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:37.637614 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.637576 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qx5hf" event={"ID":"adc13ec9-77dd-4be6-b0e5-fd3cd9671340","Type":"ContainerStarted","Data":"6926c02e2c653b718f5fd71e5c6d3280b5921ede94a38eaffcdd23e582a00bcd"} Jul 22 12:41:37.638752 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.638726 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jglnd" event={"ID":"a71bf077-f306-4681-8f97-614a73c15e41","Type":"ContainerStarted","Data":"441ae8b5aa1df92f6965e51ffc8b36b734201b0021d297b9a148cd9f3e586e2c"} Jul 22 12:41:37.641310 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.641288 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="d230dec37d583a36c461551ac8bd9ac5a44a15adcbae864fd5b37a577d8495a7" exitCode=0 Jul 22 12:41:37.641401 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.641322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"d230dec37d583a36c461551ac8bd9ac5a44a15adcbae864fd5b37a577d8495a7"} Jul 22 12:41:37.661275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.661240 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:37.826036 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.826001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:37.826036 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:37.826023 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:37.826270 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:37.826087 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls podName:ca97e449-ed09-4e0b-81af-871d3951b7ca nodeName:}" failed. No retries permitted until 2026-07-22 12:41:39.82606796 +0000 UTC m=+38.913557121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls") pod "dns-default-gk4nb" (UID: "ca97e449-ed09-4e0b-81af-871d3951b7ca") : secret "dns-default-metrics-tls" not found Jul 22 12:41:37.926886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.926799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:37.927076 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:37.926976 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.927076 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.927034 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:37.927076 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:37.927056 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:39.927034607 +0000 UTC m=+39.014523777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.927677 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:37.927649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:38.646920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:38.646889 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a83f93a-e019-4b2c-8fcb-ba754ae39cba" containerID="cda41cf4141afe6c58302efd7f62ddfb40d4de511b7df5d87260ebbda64b0346" exitCode=0 Jul 22 12:41:38.647272 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:38.646929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerDied","Data":"cda41cf4141afe6c58302efd7f62ddfb40d4de511b7df5d87260ebbda64b0346"} Jul 22 12:41:39.843576 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:39.843533 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:39.844056 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:39.843732 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:39.844056 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:39.843805 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls podName:ca97e449-ed09-4e0b-81af-871d3951b7ca nodeName:}" failed. No retries permitted until 2026-07-22 12:41:43.843785739 +0000 UTC m=+42.931274902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls") pod "dns-default-gk4nb" (UID: "ca97e449-ed09-4e0b-81af-871d3951b7ca") : secret "dns-default-metrics-tls" not found Jul 22 12:41:39.944089 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:39.944051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:39.944259 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:39.944199 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:39.944310 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:39.944267 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:43.944251801 +0000 UTC m=+43.031740961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:40.655646 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.655408 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ghswk" event={"ID":"1a83f93a-e019-4b2c-8fcb-ba754ae39cba","Type":"ContainerStarted","Data":"98c5d74bb59ed619f0f789de59c14ee333b7488899de4439727aba1b28312b95"} Jul 22 12:41:40.656887 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.656858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qx5hf" event={"ID":"adc13ec9-77dd-4be6-b0e5-fd3cd9671340","Type":"ContainerStarted","Data":"0b492e809a55a8f6c6f6ea2ff26ef577353c889279ecc74464e36d89b7c8c6a6"} Jul 22 12:41:40.658165 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.658144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jglnd" event={"ID":"a71bf077-f306-4681-8f97-614a73c15e41","Type":"ContainerStarted","Data":"4a2ae92589563f2096fb3420d200595497d8857963a81fb1b651f7dabab853bd"} Jul 22 12:41:40.684025 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.683931 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ghswk" podStartSLOduration=10.486451571 podStartE2EDuration="39.683916101s" totalImagesPullingTime="29.19746453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:40.682979763 +0000 UTC m=+39.770468941" watchObservedRunningTime="2026-07-22 12:41:40.683916101 +0000 UTC m=+39.771405274" Jul 22 12:41:40.701635 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.701580 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jglnd" podStartSLOduration=1.6097553150000001 podStartE2EDuration="4.701565039s" totalImagesPullingTime="3.091809724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:40.701168155 +0000 UTC m=+39.788657337" watchObservedRunningTime="2026-07-22 12:41:40.701565039 +0000 UTC m=+39.789054219" Jul 22 12:41:40.717163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.717108 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-qx5hf" podStartSLOduration=1.629036251 podStartE2EDuration="4.717095732s" totalImagesPullingTime="3.088059481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:40.716367509 +0000 UTC m=+39.803856690" watchObservedRunningTime="2026-07-22 12:41:40.717095732 +0000 UTC m=+39.804584913" Jul 22 12:41:40.941626 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:40.941533 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.370695 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.370619 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.491363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.491332 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.586089 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.586054 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.748393 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.748344 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.856687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.856641 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:41.999967 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:41.999875 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:42.105033 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:42.104997 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:42.200341 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:42.200301 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:42.296239 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:42.296136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:42.417806 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:42.417768 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:43.035026 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:43.034991 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:43.635275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:43.635245 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:43.877739 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:43.877680 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:43.877909 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:43.877840 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:43.877909 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:43.877906 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls podName:ca97e449-ed09-4e0b-81af-871d3951b7ca nodeName:}" failed. No retries permitted until 2026-07-22 12:41:51.877889854 +0000 UTC m=+50.965379013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls") pod "dns-default-gk4nb" (UID: "ca97e449-ed09-4e0b-81af-871d3951b7ca") : secret "dns-default-metrics-tls" not found Jul 22 12:41:43.879626 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:43.879598 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:43.978606 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:43.978564 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:43.978815 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:43.978745 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:43.978860 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:41:43.978816 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls podName:8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:51.978800859 +0000 UTC m=+51.066290019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cx4xd" (UID: "8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:44.219570 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:44.219530 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:44.421967 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:44.421871 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:44.614803 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:44.614766 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:44.827850 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:44.827812 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:45.018530 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:45.018493 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:45.349915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:45.349870 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:45.614601 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:45.614506 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:46.012368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:46.012338 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:46.118653 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:46.118615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:46.418898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:46.418809 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:46.617114 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:46.617073 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:47.046930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:47.046888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:47.541099 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:47.541064 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:47.641827 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:47.641793 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:48.288006 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:48.287971 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:48.417880 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:48.417840 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:48.621379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:48.621285 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:48.831109 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:48.831074 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:49.613211 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:49.613179 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:49.829486 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:49.829449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:50.207617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:50.207583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:50.504221 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:50.504133 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:50.605688 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:50.605651 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:50.799056 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:50.798960 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:50.991161 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:50.991129 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.091774 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.091678 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.191655 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.191621 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.250509 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.250472 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.375144 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.375056 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.577869 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.577833 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.681697 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.681604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.809172 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.809136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:51.942361 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.942334 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:51.946627 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:51.946599 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca97e449-ed09-4e0b-81af-871d3951b7ca-metrics-tls\") pod \"dns-default-gk4nb\" (UID: \"ca97e449-ed09-4e0b-81af-871d3951b7ca\") " pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:52.021082 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.021053 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:52.043102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.043064 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:52.045501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.045477 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cx4xd\" (UID: \"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2\") " pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:52.050483 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.050455 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:52.070668 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.070634 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cx4xd" Jul 22 12:41:52.204162 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.204080 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:52.233190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.231035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gk4nb"] Jul 22 12:41:52.236495 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:52.236461 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca97e449_ed09_4e0b_81af_871d3951b7ca.slice/crio-0bf2065f04b89bc74cbeb9865a441280557e6df641babe00fd48beb0f6d1881a WatchSource:0}: Error finding container 0bf2065f04b89bc74cbeb9865a441280557e6df641babe00fd48beb0f6d1881a: Status 404 returned error can't find the container with id 0bf2065f04b89bc74cbeb9865a441280557e6df641babe00fd48beb0f6d1881a Jul 22 12:41:52.247672 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.247638 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cx4xd"] Jul 22 12:41:52.250873 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:41:52.250844 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f43a315_e3a6_4ed9_9c3d_fbcf275fe4e2.slice/crio-3af26310f1ba1e094839f0018559f1d8784357f5c17e7f49300819e10adfdf9c WatchSource:0}: Error finding container 3af26310f1ba1e094839f0018559f1d8784357f5c17e7f49300819e10adfdf9c: Status 404 returned error can't find the container with id 3af26310f1ba1e094839f0018559f1d8784357f5c17e7f49300819e10adfdf9c Jul 22 12:41:52.417538 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.417504 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:52.474084 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.474000 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:52.688883 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.688852 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gk4nb" event={"ID":"ca97e449-ed09-4e0b-81af-871d3951b7ca","Type":"ContainerStarted","Data":"0bf2065f04b89bc74cbeb9865a441280557e6df641babe00fd48beb0f6d1881a"} Jul 22 12:41:52.689952 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:52.689932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx4xd" event={"ID":"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2","Type":"ContainerStarted","Data":"3af26310f1ba1e094839f0018559f1d8784357f5c17e7f49300819e10adfdf9c"} Jul 22 12:41:53.223949 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:53.223918 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:53.300156 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:53.299848 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:54.697471 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:54.697424 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gk4nb" event={"ID":"ca97e449-ed09-4e0b-81af-871d3951b7ca","Type":"ContainerStarted","Data":"f661a27e9f344532c824e2fbd0be92fb929faa845a47f8e7773cd081bb0ebbb3"} Jul 22 12:41:54.697956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:54.697487 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gk4nb" event={"ID":"ca97e449-ed09-4e0b-81af-871d3951b7ca","Type":"ContainerStarted","Data":"829d425c3ae2051c06ba4f217f86a3f6e864fe9b79673bb29b142682b8176f9a"} Jul 22 12:41:54.697956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:54.697577 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gk4nb" Jul 22 12:41:54.717211 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:54.717140 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gk4nb" podStartSLOduration=16.583940276 podStartE2EDuration="18.717119312s" totalImagesPullingTime="2.133179036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:54.715471044 +0000 UTC m=+53.802960226" watchObservedRunningTime="2026-07-22 12:41:54.717119312 +0000 UTC m=+53.804608501" Jul 22 12:41:57.232627 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:57.231467 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:57.293039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:57.293000 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:58.709837 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.709796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx4xd" event={"ID":"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2","Type":"ContainerStarted","Data":"cd34573c2357a69a6119e406b089a9098f790e349792407f44c3b2888c68cff4"} Jul 22 12:41:58.830087 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.830053 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9"] Jul 22 12:41:58.832632 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.832093 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:58.847899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.847878 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 12:41:58.848494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.848471 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jvbrg\"" Jul 22 12:41:58.851493 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.851462 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 12:41:58.868638 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.868605 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9"] Jul 22 12:41:58.908271 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.908241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcaea545-98bc-499d-b933-acaabd88910d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:58.908463 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.908279 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcaea545-98bc-499d-b933-acaabd88910d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:58.920191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.920151 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-g59c5"] Jul 22 12:41:58.922162 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.922138 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:41:58.928297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.928205 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 12:41:58.928297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.928290 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-js7b8\"" Jul 22 12:41:58.928767 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.928749 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 12:41:58.948215 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:58.948178 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-g59c5"] Jul 22 12:41:59.009586 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.009500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcaea545-98bc-499d-b933-acaabd88910d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:59.009586 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.009539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcaea545-98bc-499d-b933-acaabd88910d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:59.009857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.009610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmkz\" (UniqueName: \"kubernetes.io/projected/529542b2-c375-4db7-a47d-a91fc2eb41a8-kube-api-access-fdmkz\") pod \"downloads-5bc67dcd46-g59c5\" (UID: \"529542b2-c375-4db7-a47d-a91fc2eb41a8\") " pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:41:59.010335 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.010276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcaea545-98bc-499d-b933-acaabd88910d-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:59.012371 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.012352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fcaea545-98bc-499d-b933-acaabd88910d-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sdlx9\" (UID: \"fcaea545-98bc-499d-b933-acaabd88910d\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:59.030873 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.030838 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf68bf6f4-l8crz"] Jul 22 12:41:59.032928 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.032907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.035599 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.035573 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 12:41:59.036621 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.036601 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 12:41:59.036744 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.036633 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 12:41:59.036816 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.036781 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgszx\"" Jul 22 12:41:59.040540 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.040516 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:59.082611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.082574 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 12:41:59.104554 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.104520 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf68bf6f4-l8crz"] Jul 22 12:41:59.110524 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-tls\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-installation-pull-secrets\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmkz\" (UniqueName: \"kubernetes.io/projected/529542b2-c375-4db7-a47d-a91fc2eb41a8-kube-api-access-fdmkz\") pod \"downloads-5bc67dcd46-g59c5\" (UID: \"529542b2-c375-4db7-a47d-a91fc2eb41a8\") " pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:41:59.110669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110628 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f1a1005-c4b8-47e2-912d-d5ede38699aa-ca-trust-extracted\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110727 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-image-registry-private-configuration\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110768 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-bound-sa-token\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110786 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8z7\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-kube-api-access-fm8z7\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-trusted-ca\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.110984 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.110834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-certificates\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.143542 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.143503 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" Jul 22 12:41:59.155233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.155176 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmkz\" (UniqueName: \"kubernetes.io/projected/529542b2-c375-4db7-a47d-a91fc2eb41a8-kube-api-access-fdmkz\") pod \"downloads-5bc67dcd46-g59c5\" (UID: \"529542b2-c375-4db7-a47d-a91fc2eb41a8\") " pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:41:59.212122 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212084 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-tls\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.212282 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-installation-pull-secrets\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f1a1005-c4b8-47e2-912d-d5ede38699aa-ca-trust-extracted\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212513 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-image-registry-private-configuration\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-bound-sa-token\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8z7\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-kube-api-access-fm8z7\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-trusted-ca\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.212664 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-certificates\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.213386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f1a1005-c4b8-47e2-912d-d5ede38699aa-ca-trust-extracted\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.213880 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.213688 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-certificates\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.215052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.215027 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f1a1005-c4b8-47e2-912d-d5ede38699aa-trusted-ca\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.216741 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.216676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-installation-pull-secrets\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.217306 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.217262 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-registry-tls\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.218587 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.218544 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f1a1005-c4b8-47e2-912d-d5ede38699aa-image-registry-private-configuration\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.223459 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.223433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-bound-sa-token\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.225700 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.225675 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8z7\" (UniqueName: \"kubernetes.io/projected/8f1a1005-c4b8-47e2-912d-d5ede38699aa-kube-api-access-fm8z7\") pod \"image-registry-5cf68bf6f4-l8crz\" (UID: \"8f1a1005-c4b8-47e2-912d-d5ede38699aa\") " pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.232632 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.232509 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:41:59.284242 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.284163 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:59.309302 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.309227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9"] Jul 22 12:41:59.344271 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.344214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:41:59.388964 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.388245 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-g59c5"] Jul 22 12:41:59.395524 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.395485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:59.497819 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.497770 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:59.627347 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.627268 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:59.690831 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:41:59.690776 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:00.757584 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:00.757536 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaea545_98bc_499d_b933_acaabd88910d.slice/crio-c5cf5f7670e8ac99e227e960e369e6e25e2b149d0301879024dcc59019f62d11 WatchSource:0}: Error finding container c5cf5f7670e8ac99e227e960e369e6e25e2b149d0301879024dcc59019f62d11: Status 404 returned error can't find the container with id c5cf5f7670e8ac99e227e960e369e6e25e2b149d0301879024dcc59019f62d11 Jul 22 12:42:00.758150 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:00.758120 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529542b2_c375_4db7_a47d_a91fc2eb41a8.slice/crio-e2f395003cee5afd41f6b15e9d5a227918c73c64d70fe0bae87c91f357441365 WatchSource:0}: Error finding container e2f395003cee5afd41f6b15e9d5a227918c73c64d70fe0bae87c91f357441365: Status 404 returned error can't find the container with id e2f395003cee5afd41f6b15e9d5a227918c73c64d70fe0bae87c91f357441365 Jul 22 12:42:01.406829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.406780 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:01.520544 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.520505 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:01.580617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.580585 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf68bf6f4-l8crz"] Jul 22 12:42:01.590319 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:01.589884 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1a1005_c4b8_47e2_912d_d5ede38699aa.slice/crio-728c8d44c24962745ed59371ac0b222b8574a96feb65afe236aa2210f3eba691 WatchSource:0}: Error finding container 728c8d44c24962745ed59371ac0b222b8574a96feb65afe236aa2210f3eba691: Status 404 returned error can't find the container with id 728c8d44c24962745ed59371ac0b222b8574a96feb65afe236aa2210f3eba691 Jul 22 12:42:01.720003 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.719896 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-g59c5" event={"ID":"529542b2-c375-4db7-a47d-a91fc2eb41a8","Type":"ContainerStarted","Data":"e2f395003cee5afd41f6b15e9d5a227918c73c64d70fe0bae87c91f357441365"} Jul 22 12:42:01.722243 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.722203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx4xd" event={"ID":"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2","Type":"ContainerStarted","Data":"eb6e1445bd8daa12196d88a5d8107f51539bce24b17b68febc58c64624489143"} Jul 22 12:42:01.724269 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.724235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" event={"ID":"8f1a1005-c4b8-47e2-912d-d5ede38699aa","Type":"ContainerStarted","Data":"c0a39679195ea724f0887e8405884d8a2ddd5e57e9d1fe23d8f5298e9246391d"} Jul 22 12:42:01.724421 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.724288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" event={"ID":"8f1a1005-c4b8-47e2-912d-d5ede38699aa","Type":"ContainerStarted","Data":"728c8d44c24962745ed59371ac0b222b8574a96feb65afe236aa2210f3eba691"} Jul 22 12:42:01.724738 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.724674 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:42:01.725734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.725666 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" event={"ID":"fcaea545-98bc-499d-b933-acaabd88910d","Type":"ContainerStarted","Data":"c5cf5f7670e8ac99e227e960e369e6e25e2b149d0301879024dcc59019f62d11"} Jul 22 12:42:01.746556 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:01.746472 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" podStartSLOduration=3.746452851 podStartE2EDuration="3.746452851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:01.745485209 +0000 UTC m=+60.832974390" watchObservedRunningTime="2026-07-22 12:42:01.746452851 +0000 UTC m=+60.833942031" Jul 22 12:42:02.731858 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:02.731811 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" event={"ID":"fcaea545-98bc-499d-b933-acaabd88910d","Type":"ContainerStarted","Data":"a2aaab9b2a13b0c58104289323d8226fffd49d0acc2e163d996f4c909bf18c05"} Jul 22 12:42:02.750985 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:02.750928 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sdlx9" podStartSLOduration=3.239752365 podStartE2EDuration="4.750912779s" totalImagesPullingTime="1.511160414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:02.750195112 +0000 UTC m=+61.837684293" watchObservedRunningTime="2026-07-22 12:42:02.750912779 +0000 UTC m=+61.838401954" Jul 22 12:42:02.904493 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:02.904442 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:02.993996 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:02.993873 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.443201 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.443104 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.529771 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.529730 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.627520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.627485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.640426 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.640381 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srz7w" Jul 22 12:42:03.728521 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.728417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.833959 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.833920 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:03.937618 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:03.937572 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:04.288774 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:04.288736 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:04.390530 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:04.390492 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:04.702811 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:04.702782 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gk4nb" Jul 22 12:42:05.520237 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:05.520199 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:05.622487 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:05.622447 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:05.720501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:05.720454 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:05.824836 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:05.824763 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:05.925476 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:05.925448 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:06.748218 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:06.748185 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cx4xd" event={"ID":"8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2","Type":"ContainerStarted","Data":"3cedace291107ee8898f208a45882bddbf46cee4f0c2129f790b8c44c0db2d61"} Jul 22 12:42:06.770002 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:06.769941 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cx4xd" podStartSLOduration=17.301442998 podStartE2EDuration="30.769919101s" totalImagesPullingTime="13.468476103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:06.768667482 +0000 UTC m=+65.856156665" watchObservedRunningTime="2026-07-22 12:42:06.769919101 +0000 UTC m=+65.857408278" Jul 22 12:42:07.285018 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.284963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:42:07.285220 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.285036 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:42:07.285220 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.285071 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:42:07.288036 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.288006 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:42:07.288199 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.288049 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:42:07.288587 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.288558 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:07.288829 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.288803 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 12:42:07.298128 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.298092 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 12:42:07.298317 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.298297 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bede90bf-2533-4511-a623-8e219f89fefc-metrics-certs\") pod \"network-metrics-daemon-f8p8r\" (UID: \"bede90bf-2533-4511-a623-8e219f89fefc\") " pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:42:07.298579 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.298553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf7e899a-13ae-4fa5-a1be-4677dbf43f4a-original-pull-secret\") pod \"global-pull-secret-syncer-jxnk6\" (UID: \"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a\") " pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:42:07.308973 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.308947 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxdz\" (UniqueName: \"kubernetes.io/projected/56f83998-325d-4173-be07-bfce1b02e5c4-kube-api-access-kpxdz\") pod \"network-check-target-zmhb7\" (UID: \"56f83998-325d-4173-be07-bfce1b02e5c4\") " pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:42:07.386914 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.386880 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:07.390584 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.390558 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxnk6" Jul 22 12:42:07.496495 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.496465 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:42:07.501131 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.501103 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:42:07.503678 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.503642 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:42:07.508795 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.508770 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f8p8r" Jul 22 12:42:07.555196 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.555151 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxnk6"] Jul 22 12:42:07.558401 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:07.558335 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7e899a_13ae_4fa5_a1be_4677dbf43f4a.slice/crio-6be092dc5c8008dd51d5ae9807a78f9218df022b168451537dd7048b198e7044 WatchSource:0}: Error finding container 6be092dc5c8008dd51d5ae9807a78f9218df022b168451537dd7048b198e7044: Status 404 returned error can't find the container with id 6be092dc5c8008dd51d5ae9807a78f9218df022b168451537dd7048b198e7044 Jul 22 12:42:07.693892 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.693801 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zmhb7"] Jul 22 12:42:07.696951 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:07.696916 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f83998_325d_4173_be07_bfce1b02e5c4.slice/crio-ff09321870b4b4c86fdaacd27bf06bde25ed39ec1d3f037cf0cef5d318937d26 WatchSource:0}: Error finding container ff09321870b4b4c86fdaacd27bf06bde25ed39ec1d3f037cf0cef5d318937d26: Status 404 returned error can't find the container with id ff09321870b4b4c86fdaacd27bf06bde25ed39ec1d3f037cf0cef5d318937d26 Jul 22 12:42:07.723416 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.723381 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f8p8r"] Jul 22 12:42:07.727150 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:07.727122 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede90bf_2533_4511_a623_8e219f89fefc.slice/crio-421e1bf20e050920b932795d77ade10f2d5cbb8cfea66cd7692fd8706fee5244 WatchSource:0}: Error finding container 421e1bf20e050920b932795d77ade10f2d5cbb8cfea66cd7692fd8706fee5244: Status 404 returned error can't find the container with id 421e1bf20e050920b932795d77ade10f2d5cbb8cfea66cd7692fd8706fee5244 Jul 22 12:42:07.753517 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.753450 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8p8r" event={"ID":"bede90bf-2533-4511-a623-8e219f89fefc","Type":"ContainerStarted","Data":"421e1bf20e050920b932795d77ade10f2d5cbb8cfea66cd7692fd8706fee5244"} Jul 22 12:42:07.754873 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.754840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmhb7" event={"ID":"56f83998-325d-4173-be07-bfce1b02e5c4","Type":"ContainerStarted","Data":"ff09321870b4b4c86fdaacd27bf06bde25ed39ec1d3f037cf0cef5d318937d26"} Jul 22 12:42:07.756135 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.756110 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxnk6" event={"ID":"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a","Type":"ContainerStarted","Data":"6be092dc5c8008dd51d5ae9807a78f9218df022b168451537dd7048b198e7044"} Jul 22 12:42:07.917241 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:07.917101 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:08.749181 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:08.749121 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:08.841975 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:08.841831 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:09.161636 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.161574 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:09.246276 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.245178 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:09.248687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.247644 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.251675 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.251925 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.252132 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.252335 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-22jtw\"" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.252545 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 12:42:09.253549 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.252745 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 12:42:09.260615 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.259617 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:09.298686 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298647 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmrj\" (UniqueName: \"kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.298882 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.298882 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298841 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.298998 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298889 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.298998 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.298998 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.298953 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400258 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400164 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmrj\" (UniqueName: \"kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400258 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400215 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400258 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400533 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400267 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400533 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.400533 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.400325 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.401252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.401227 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.401602 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.401576 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.401697 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.401617 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.404793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.403973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.404793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.404027 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.416843 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.416782 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmrj\" (UniqueName: \"kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj\") pod \"console-57bc45ccc5-x2r6d\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.571871 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.571591 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:09.755236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.755145 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:09.757651 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:09.757612 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9e0b17_5bc0_45ec_b1ba_b189bd4f4ec1.slice/crio-3903efaa4b0189a7f538446b027fe64b7092c9633ed6bde45bfb8a6b3772b34a WatchSource:0}: Error finding container 3903efaa4b0189a7f538446b027fe64b7092c9633ed6bde45bfb8a6b3772b34a: Status 404 returned error can't find the container with id 3903efaa4b0189a7f538446b027fe64b7092c9633ed6bde45bfb8a6b3772b34a Jul 22 12:42:09.770612 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.770063 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8p8r" event={"ID":"bede90bf-2533-4511-a623-8e219f89fefc","Type":"ContainerStarted","Data":"ea608e15921b82a07825e1e4d828eabd4031d16545e6da4f0f0c2922a2badfb6"} Jul 22 12:42:09.770612 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.770136 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f8p8r" event={"ID":"bede90bf-2533-4511-a623-8e219f89fefc","Type":"ContainerStarted","Data":"0f17ca88d84b564960435f5b35639c497fbc39c40678534caba75c83c532d226"} Jul 22 12:42:09.772196 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.772126 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bc45ccc5-x2r6d" event={"ID":"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1","Type":"ContainerStarted","Data":"3903efaa4b0189a7f538446b027fe64b7092c9633ed6bde45bfb8a6b3772b34a"} Jul 22 12:42:09.789022 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.788957 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f8p8r" podStartSLOduration=67.413309276 podStartE2EDuration="1m8.788933896s" totalImagesPullingTime="1.37562462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:09.788347276 +0000 UTC m=+68.875836457" watchObservedRunningTime="2026-07-22 12:42:09.788933896 +0000 UTC m=+68.876423079" Jul 22 12:42:09.906855 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.906698 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:09.980448 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:09.980256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:11.031745 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:11.031645 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:11.134935 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:11.134846 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:11.438062 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:11.437888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:12.190312 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:12.190274 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:12.291504 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:12.291454 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:12.925989 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:12.925937 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.132384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.132350 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.182940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.182639 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mvskl"] Jul 22 12:42:13.192515 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.192468 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.195596 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.195566 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 12:42:13.196212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.196049 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 12:42:13.196212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.196181 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 12:42:13.198214 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.198193 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 12:42:13.198525 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.198493 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 12:42:13.198725 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.198666 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2xzrv\"" Jul 22 12:42:13.198804 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.198362 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 12:42:13.265108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.265074 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.333223 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333181 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-textfile\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333293 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-wtmp\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333354 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-sys\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333439 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl74\" (UniqueName: \"kubernetes.io/projected/29d46ac7-b021-4a53-af43-1558cdd10349-kube-api-access-7zl74\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-metrics-client-ca\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-root\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.333642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.333563 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-textfile\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434144 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434182 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-wtmp\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-sys\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434271 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl74\" (UniqueName: \"kubernetes.io/projected/29d46ac7-b021-4a53-af43-1558cdd10349-kube-api-access-7zl74\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-metrics-client-ca\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-root\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434364 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434423 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-textfile\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-sys\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.434991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-wtmp\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:42:13.435083 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:42:13.435139 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls podName:29d46ac7-b021-4a53-af43-1558cdd10349 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:13.935119376 +0000 UTC m=+73.022608543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls") pod "node-exporter-mvskl" (UID: "29d46ac7-b021-4a53-af43-1558cdd10349") : secret "node-exporter-tls" not found Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.435181 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d46ac7-b021-4a53-af43-1558cdd10349-root\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.435680 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.437007 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.436091 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d46ac7-b021-4a53-af43-1558cdd10349-metrics-client-ca\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.438880 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.438826 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.444608 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.444545 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl74\" (UniqueName: \"kubernetes.io/projected/29d46ac7-b021-4a53-af43-1558cdd10349-kube-api-access-7zl74\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.454336 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.454258 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.554180 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.554131 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.648743 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.648689 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.746228 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.746172 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:13.939377 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.939333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:13.942423 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:13.942341 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d46ac7-b021-4a53-af43-1558cdd10349-node-exporter-tls\") pod \"node-exporter-mvskl\" (UID: \"29d46ac7-b021-4a53-af43-1558cdd10349\") " pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:14.066381 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.066279 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:14.107308 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.107241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvskl" Jul 22 12:42:14.236531 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.236488 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:14.333355 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.333264 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:14.362639 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.362597 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:42:14.405503 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.405459 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:42:14.405695 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.405616 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.431330 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.431293 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 12:42:14.431527 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.431299 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 12:42:14.431527 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.431307 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 12:42:14.431981 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.431962 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cckpq\"" Jul 22 12:42:14.432079 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.431965 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 12:42:14.432470 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.432453 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 12:42:14.432773 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.432748 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 12:42:14.435664 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.435643 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 12:42:14.437060 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.437041 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 12:42:14.438738 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.438702 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 12:42:14.441828 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.441785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:14.544389 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544389 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544394 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544430 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544563 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544650 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw9w\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544782 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.544857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544809 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.545046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544894 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.545046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.544921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.604502 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.604407 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:14.646143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646111 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646225 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646266 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw9w\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646296 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646660 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646600 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646660 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646652 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646790 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646790 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646781 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.646930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.646853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.647384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.647206 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.648306 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.647991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649042 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.648990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.649455 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.649488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.649596 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649852 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.649673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.649919 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.649847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.650500 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.650479 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.650865 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.650841 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.651034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.651017 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.651182 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.651167 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.659989 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.659963 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw9w\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w\") pod \"alertmanager-main-0\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:14.716898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:14.716859 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:42:15.068216 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:15.068169 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:15.169853 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:15.169757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.166059 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.166019 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.271218 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.271177 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.406561 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.406507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.484143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.484071 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.717345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.717305 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.884403 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.884281 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:16.948455 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:16.948410 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:17.231745 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:17.231685 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:17.312935 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:17.312894 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:17.404831 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:17.404764 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:17.498013 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:17.497918 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:18.003318 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.003276 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:18.057232 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.057150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:18.112954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.112913 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:18.205372 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.205322 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:18.232076 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.232031 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:18.232539 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.232190 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.242604 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.242561 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 12:42:18.380484 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380484 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380452 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxx6h\" (UniqueName: \"kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380484 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380754 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380511 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380754 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380754 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.380754 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.380724 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.441152 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.441120 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:18.445658 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.445627 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67cc54c98f-zg7gk"] Jul 22 12:42:18.482163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxx6h\" (UniqueName: \"kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482439 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.482480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482467 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.483008 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.482979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.483131 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.483063 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.483507 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.483459 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.483600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.483552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.485546 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.485520 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.485730 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.485686 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.487760 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.487731 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67cc54c98f-zg7gk"] Jul 22 12:42:18.487887 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.487845 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.490646 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.490619 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 12:42:18.491396 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.491369 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 12:42:18.491517 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.491489 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 12:42:18.491593 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.491526 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2zvxf\"" Jul 22 12:42:18.491687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.491672 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 12:42:18.491970 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.491913 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxx6h\" (UniqueName: \"kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h\") pod \"console-67f67fbbcb-fq6wp\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.492213 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.492194 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 12:42:18.498684 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.498642 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 12:42:18.546592 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.546541 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:18.583052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-metrics-client-ca\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-federate-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583143 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583193 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llt44\" (UniqueName: \"kubernetes.io/projected/0d97a441-7fe1-455f-a164-523e7a67be35-kube-api-access-llt44\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583434 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583240 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583434 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-serving-certs-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.583434 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.583302 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684609 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684524 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-metrics-client-ca\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684609 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684583 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-federate-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684609 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llt44\" (UniqueName: \"kubernetes.io/projected/0d97a441-7fe1-455f-a164-523e7a67be35-kube-api-access-llt44\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684743 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-serving-certs-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.684920 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.684831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.685373 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.685348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-metrics-client-ca\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.685632 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.685581 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.685768 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.685738 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d97a441-7fe1-455f-a164-523e7a67be35-serving-certs-ca-bundle\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.687765 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.687737 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.687887 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.687852 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-secret-telemeter-client\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.687953 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.687928 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-telemeter-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.687953 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.687942 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d97a441-7fe1-455f-a164-523e7a67be35-federate-client-tls\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.698168 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.698132 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llt44\" (UniqueName: \"kubernetes.io/projected/0d97a441-7fe1-455f-a164-523e7a67be35-kube-api-access-llt44\") pod \"telemeter-client-67cc54c98f-zg7gk\" (UID: \"0d97a441-7fe1-455f-a164-523e7a67be35\") " pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:18.812049 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:18.812007 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" Jul 22 12:42:19.523255 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.523220 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:19.524639 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.524607 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:42:19.569944 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.569886 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:42:19.570120 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.570054 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.572556 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.572527 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 12:42:19.574916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.574679 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 12:42:19.574916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.574721 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 12:42:19.574916 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.574873 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 12:42:19.575381 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575364 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 12:42:19.575940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575663 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 12:42:19.575940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575695 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 12:42:19.575940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575739 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 12:42:19.575940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575771 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 12:42:19.575940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.575812 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 12:42:19.576261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.576213 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dhujsck446n6c\"" Jul 22 12:42:19.576842 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.576817 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 12:42:19.576963 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.576863 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nq5x\"" Jul 22 12:42:19.577162 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.577146 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 12:42:19.577471 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.577452 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 12:42:19.578472 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.578419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 12:42:19.583917 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.583892 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 12:42:19.695253 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695253 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695257 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695318 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695338 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695365 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695478 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695535 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxmt\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695555 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695589 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695613 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695664 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695754 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695844 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.695915 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.695868 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797063 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxmt\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797185 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.797453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.797325 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798058 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798277 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798302 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798365 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798463 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798523 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.798957 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.798618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.799622 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.799006 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.799889 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.799865 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.800235 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.800207 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.800420 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.800396 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.800844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.800496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.802073 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.801739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.802308 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.802282 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.802812 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.802786 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.802894 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.802872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.802973 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.802933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.803371 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.803346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.803585 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.803560 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.804814 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.804788 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.805038 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.804896 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.805038 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.804909 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.805327 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.805307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.807979 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.807960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxmt\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt\") pod \"prometheus-k8s-0\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:19.883362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:19.883312 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:22.358152 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:22.358121 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d46ac7_b021_4a53_af43_1558cdd10349.slice/crio-73a68f276a17ffe28b45ca5f0c1a0b070d94484a9135bf294bf37d266a82fe22 WatchSource:0}: Error finding container 73a68f276a17ffe28b45ca5f0c1a0b070d94484a9135bf294bf37d266a82fe22: Status 404 returned error can't find the container with id 73a68f276a17ffe28b45ca5f0c1a0b070d94484a9135bf294bf37d266a82fe22 Jul 22 12:42:22.623013 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.622970 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:42:22.627930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.627893 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:22.633251 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:22.633207 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce14185_505c_4333_bcee_46362a771187.slice/crio-0582cb7c59e1ae2667f1076c29f6eed15cc96a36bf1f31f105c282421907cae8 WatchSource:0}: Error finding container 0582cb7c59e1ae2667f1076c29f6eed15cc96a36bf1f31f105c282421907cae8: Status 404 returned error can't find the container with id 0582cb7c59e1ae2667f1076c29f6eed15cc96a36bf1f31f105c282421907cae8 Jul 22 12:42:22.634758 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:22.634704 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa370ef_944d_4957_9c7f_b44cb77cd0b0.slice/crio-5c93d6d41599bb60e53e05251a5df1112b4147c27f7ed04f2a967353b27b65c6 WatchSource:0}: Error finding container 5c93d6d41599bb60e53e05251a5df1112b4147c27f7ed04f2a967353b27b65c6: Status 404 returned error can't find the container with id 5c93d6d41599bb60e53e05251a5df1112b4147c27f7ed04f2a967353b27b65c6 Jul 22 12:42:22.662014 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.661658 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:42:22.664413 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:22.664377 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a09bbd3_d78b_45a4_a508_b7a18f2d9e8d.slice/crio-992a6a3b6daea72c88260561debe257e889d90435e943768aaa2fde6bd483bf8 WatchSource:0}: Error finding container 992a6a3b6daea72c88260561debe257e889d90435e943768aaa2fde6bd483bf8: Status 404 returned error can't find the container with id 992a6a3b6daea72c88260561debe257e889d90435e943768aaa2fde6bd483bf8 Jul 22 12:42:22.737651 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.737617 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf68bf6f4-l8crz" Jul 22 12:42:22.841389 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.841161 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f67fbbcb-fq6wp" event={"ID":"7aa370ef-944d-4957-9c7f-b44cb77cd0b0","Type":"ContainerStarted","Data":"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a"} Jul 22 12:42:22.841389 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.841226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f67fbbcb-fq6wp" event={"ID":"7aa370ef-944d-4957-9c7f-b44cb77cd0b0","Type":"ContainerStarted","Data":"5c93d6d41599bb60e53e05251a5df1112b4147c27f7ed04f2a967353b27b65c6"} Jul 22 12:42:22.844228 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.844192 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"0582cb7c59e1ae2667f1076c29f6eed15cc96a36bf1f31f105c282421907cae8"} Jul 22 12:42:22.846854 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.846769 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-g59c5" event={"ID":"529542b2-c375-4db7-a47d-a91fc2eb41a8","Type":"ContainerStarted","Data":"b00e14a104a0593d2c6f28a2231fae71fa57711683b63678bdab7ec3c28a2bd0"} Jul 22 12:42:22.847372 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.847330 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:42:22.849445 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.849419 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxnk6" event={"ID":"bf7e899a-13ae-4fa5-a1be-4677dbf43f4a","Type":"ContainerStarted","Data":"e8073e0907df46f6f60aa02f3b62141d691ee19c31ee1cd6d824e833541c1945"} Jul 22 12:42:22.851038 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.851006 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"992a6a3b6daea72c88260561debe257e889d90435e943768aaa2fde6bd483bf8"} Jul 22 12:42:22.852661 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.852598 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvskl" event={"ID":"29d46ac7-b021-4a53-af43-1558cdd10349","Type":"ContainerStarted","Data":"73a68f276a17ffe28b45ca5f0c1a0b070d94484a9135bf294bf37d266a82fe22"} Jul 22 12:42:22.855131 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.855105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zmhb7" event={"ID":"56f83998-325d-4173-be07-bfce1b02e5c4","Type":"ContainerStarted","Data":"c5639860a4f273277c6de96a614294419400dab76f92f30a3fafe87e4ff9a550"} Jul 22 12:42:22.855480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.855428 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:42:22.858504 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.858472 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bc45ccc5-x2r6d" event={"ID":"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1","Type":"ContainerStarted","Data":"a352965d9f876b00d955d06708c4abded0494cda015faf95064a76e412339b2c"} Jul 22 12:42:22.860674 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.860313 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-g59c5" Jul 22 12:42:22.872927 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.872790 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67cc54c98f-zg7gk"] Jul 22 12:42:22.874224 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.873745 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f67fbbcb-fq6wp" podStartSLOduration=4.873724018 podStartE2EDuration="4.873724018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.869182281 +0000 UTC m=+81.956671463" watchObservedRunningTime="2026-07-22 12:42:22.873724018 +0000 UTC m=+81.961213191" Jul 22 12:42:22.880585 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:42:22.880342 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d97a441_7fe1_455f_a164_523e7a67be35.slice/crio-7eef8ea2ba078a9c33aa937ebf7c014a0d7305eb2aa484242d0c3a23e9f571d1 WatchSource:0}: Error finding container 7eef8ea2ba078a9c33aa937ebf7c014a0d7305eb2aa484242d0c3a23e9f571d1: Status 404 returned error can't find the container with id 7eef8ea2ba078a9c33aa937ebf7c014a0d7305eb2aa484242d0c3a23e9f571d1 Jul 22 12:42:22.915809 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.915428 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jxnk6" podStartSLOduration=67.122428658 podStartE2EDuration="1m21.915406083s" totalImagesPullingTime="14.792977425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.889860855 +0000 UTC m=+81.977350046" watchObservedRunningTime="2026-07-22 12:42:22.915406083 +0000 UTC m=+82.002895267" Jul 22 12:42:22.916395 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.916341 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-g59c5" podStartSLOduration=3.257352049 podStartE2EDuration="24.916328736s" totalImagesPullingTime="21.658976687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.915020861 +0000 UTC m=+82.002510042" watchObservedRunningTime="2026-07-22 12:42:22.916328736 +0000 UTC m=+82.003817916" Jul 22 12:42:22.941323 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.941167 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57bc45ccc5-x2r6d" podStartSLOduration=1.347367927 podStartE2EDuration="13.941141216s" totalImagesPullingTime="12.593773289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.939366321 +0000 UTC m=+82.026855523" watchObservedRunningTime="2026-07-22 12:42:22.941141216 +0000 UTC m=+82.028630399" Jul 22 12:42:22.967383 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.967309 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zmhb7" podStartSLOduration=67.313245623 podStartE2EDuration="1m21.967286389s" totalImagesPullingTime="14.654040766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.966160627 +0000 UTC m=+82.053649810" watchObservedRunningTime="2026-07-22 12:42:22.967286389 +0000 UTC m=+82.054775572" Jul 22 12:42:22.978998 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:22.978924 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:23.109260 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:23.109223 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:23.207000 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:23.206941 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:23.866043 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:23.865526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" event={"ID":"0d97a441-7fe1-455f-a164-523e7a67be35","Type":"ContainerStarted","Data":"7eef8ea2ba078a9c33aa937ebf7c014a0d7305eb2aa484242d0c3a23e9f571d1"} Jul 22 12:42:24.872083 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:24.872030 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvskl" event={"ID":"29d46ac7-b021-4a53-af43-1558cdd10349","Type":"ContainerStarted","Data":"36d6d127ba897a8e3771213535547dee38501ff0f0fc2077f292ad687dae86e4"} Jul 22 12:42:24.874284 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:24.874153 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" exitCode=0 Jul 22 12:42:24.874284 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:24.874199 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} Jul 22 12:42:24.876694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:24.876663 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60" exitCode=0 Jul 22 12:42:24.876816 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:24.876753 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60"} Jul 22 12:42:25.565655 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.565619 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:25.632311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.632273 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:25.891191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.891061 2557 generic.go:356] "Generic (PLEG): container finished" podID="29d46ac7-b021-4a53-af43-1558cdd10349" containerID="36d6d127ba897a8e3771213535547dee38501ff0f0fc2077f292ad687dae86e4" exitCode=0 Jul 22 12:42:25.891191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.891152 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvskl" event={"ID":"29d46ac7-b021-4a53-af43-1558cdd10349","Type":"ContainerDied","Data":"36d6d127ba897a8e3771213535547dee38501ff0f0fc2077f292ad687dae86e4"} Jul 22 12:42:25.896772 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.896734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" event={"ID":"0d97a441-7fe1-455f-a164-523e7a67be35","Type":"ContainerStarted","Data":"73cec9a3dc7e484b34241b13fffc6797881ca831cec3d9343bf38c63f27d5c8a"} Jul 22 12:42:25.926588 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.926077 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:25.991113 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:25.991036 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:26.908131 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.908032 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvskl" event={"ID":"29d46ac7-b021-4a53-af43-1558cdd10349","Type":"ContainerStarted","Data":"da00d57ff5f2c274faa38d1bfd4dcfe8e63c83bae17c6624cc31692f211b638c"} Jul 22 12:42:26.908131 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.908095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvskl" event={"ID":"29d46ac7-b021-4a53-af43-1558cdd10349","Type":"ContainerStarted","Data":"ed6e8d79a56f0c55b747810faa31479277bdf814fd3e79cb26be613a88543f5c"} Jul 22 12:42:26.911674 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.911640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" event={"ID":"0d97a441-7fe1-455f-a164-523e7a67be35","Type":"ContainerStarted","Data":"22f793ee62fc3ff3d675d0f982bec0a010440bb603f0356e9e499893d1b588e3"} Jul 22 12:42:26.911845 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.911729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" event={"ID":"0d97a441-7fe1-455f-a164-523e7a67be35","Type":"ContainerStarted","Data":"a461e5ade6da6851b0424ae7396e7e0dd9e55cf45ca7c9f58fea0fce5096d92a"} Jul 22 12:42:26.934359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.932417 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mvskl" podStartSLOduration=11.98256138 podStartE2EDuration="13.932392972s" totalImagesPullingTime="1.840611272s" totalInitContainerRuntime="109.22032ms" isStatefulPod=true podCreationTimestamp="2026-07-22 12:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:26.93061399 +0000 UTC m=+86.018103173" watchObservedRunningTime="2026-07-22 12:42:26.932392972 +0000 UTC m=+86.019882134" Jul 22 12:42:26.957963 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:26.957906 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67cc54c98f-zg7gk" podStartSLOduration=6.074581833 podStartE2EDuration="8.957882271s" totalImagesPullingTime="2.883300438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:26.955562575 +0000 UTC m=+86.043051757" watchObservedRunningTime="2026-07-22 12:42:26.957882271 +0000 UTC m=+86.045371452" Jul 22 12:42:27.184669 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:27.184578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:27.254465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:27.254427 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:28.078429 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:28.077724 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:28.547740 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:28.547637 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:28.717551 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:28.717465 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:28.857756 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:28.857621 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:29.572556 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.572475 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:29.930381 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.930352 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} Jul 22 12:42:29.934149 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.934122 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793"} Jul 22 12:42:29.934259 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.934182 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136"} Jul 22 12:42:29.934259 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.934208 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d"} Jul 22 12:42:29.934259 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:29.934231 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213"} Jul 22 12:42:30.947192 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:30.947087 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} Jul 22 12:42:30.953377 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:30.953344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a"} Jul 22 12:42:31.961345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:31.961308 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerStarted","Data":"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc"} Jul 22 12:42:31.994977 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:31.994912 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.603302325 podStartE2EDuration="17.994889651s" totalImagesPullingTime="6.363791544s" totalInitContainerRuntime="27.795782ms" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:31.992046277 +0000 UTC m=+91.079535458" watchObservedRunningTime="2026-07-22 12:42:31.994889651 +0000 UTC m=+91.082378836" Jul 22 12:42:32.968106 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:32.968072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} Jul 22 12:42:32.968442 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:32.968171 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} Jul 22 12:42:33.192600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:33.192569 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:33.328132 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:33.328101 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:33.975423 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:33.975392 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} Jul 22 12:42:33.975886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:33.975471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerStarted","Data":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} Jul 22 12:42:33.988564 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:33.988536 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:34.023953 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:34.023898 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.57082732 podStartE2EDuration="15.023880647s" totalImagesPullingTime="9.418321139s" totalInitContainerRuntime="34.732188ms" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:34.020260246 +0000 UTC m=+93.107749438" watchObservedRunningTime="2026-07-22 12:42:34.023880647 +0000 UTC m=+93.111369828" Jul 22 12:42:34.884148 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:34.884114 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:42:39.147582 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:39.147533 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:39.248147 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:39.248108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:39.358239 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:39.358198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:39.881442 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:39.881409 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:39.972558 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:39.972524 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:40.975734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:40.975679 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:41.064117 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:41.064082 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:42.186433 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:42.186398 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:42.287696 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:42.287650 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:42:47.887791 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:47.887742 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57bc45ccc5-x2r6d" podUID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" containerName="console" containerID="cri-o://a352965d9f876b00d955d06708c4abded0494cda015faf95064a76e412339b2c" gracePeriod=15 Jul 22 12:42:48.030849 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.030813 2557 generic.go:356] "Generic (PLEG): container finished" podID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" containerID="a352965d9f876b00d955d06708c4abded0494cda015faf95064a76e412339b2c" exitCode=0 Jul 22 12:42:48.031012 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.030873 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bc45ccc5-x2r6d" event={"ID":"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1","Type":"ContainerDied","Data":"a352965d9f876b00d955d06708c4abded0494cda015faf95064a76e412339b2c"} Jul 22 12:42:48.165693 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.165608 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:48.260575 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260538 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.260575 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260580 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.260886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260646 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnmrj\" (UniqueName: \"kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.260886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260689 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.260886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260845 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.261050 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.260895 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca\") pod \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\" (UID: \"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1\") " Jul 22 12:42:48.261146 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.261124 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:48.261261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.261179 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config" (OuterVolumeSpecName: "console-config") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:48.261346 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.261311 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca" (OuterVolumeSpecName: "service-ca") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:48.263085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.263063 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj" (OuterVolumeSpecName: "kube-api-access-jnmrj") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "kube-api-access-jnmrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:42:48.263251 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.263234 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:48.263356 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.263334 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" (UID: "cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:48.361845 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361810 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-serving-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.361845 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361839 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-oauth-serving-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.361845 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361847 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmrj\" (UniqueName: \"kubernetes.io/projected/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-kube-api-access-jnmrj\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.362072 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361861 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-oauth-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.362072 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361870 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-console-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.362072 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:48.361879 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:49.035848 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.035820 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57bc45ccc5-x2r6d" Jul 22 12:42:49.036434 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.035825 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57bc45ccc5-x2r6d" event={"ID":"cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1","Type":"ContainerDied","Data":"3903efaa4b0189a7f538446b027fe64b7092c9633ed6bde45bfb8a6b3772b34a"} Jul 22 12:42:49.036434 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.035951 2557 scope.go:117] "RemoveContainer" containerID="a352965d9f876b00d955d06708c4abded0494cda015faf95064a76e412339b2c" Jul 22 12:42:49.061314 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.061284 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:49.065166 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.065140 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57bc45ccc5-x2r6d"] Jul 22 12:42:49.484989 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:49.484963 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" path="/var/lib/kubelet/pods/cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1/volumes" Jul 22 12:42:53.105963 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.105925 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f67fbbcb-fq6wp" podUID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" containerName="console" containerID="cri-o://8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a" gracePeriod=15 Jul 22 12:42:53.386821 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.386781 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:53.504060 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504021 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504060 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504060 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504249 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504249 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504103 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504249 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504159 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxx6h\" (UniqueName: \"kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504249 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504191 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504249 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504232 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert\") pod \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\" (UID: \"7aa370ef-944d-4957-9c7f-b44cb77cd0b0\") " Jul 22 12:42:53.504534 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504511 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca" (OuterVolumeSpecName: "service-ca") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:53.504665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504640 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:53.504746 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504680 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:53.504746 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.504689 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config" (OuterVolumeSpecName: "console-config") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:53.506477 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.506450 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:53.506540 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.506472 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h" (OuterVolumeSpecName: "kube-api-access-zxx6h") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "kube-api-access-zxx6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:42:53.506540 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.506492 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7aa370ef-944d-4957-9c7f-b44cb77cd0b0" (UID: "7aa370ef-944d-4957-9c7f-b44cb77cd0b0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:53.605092 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605048 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605092 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605088 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-oauth-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605102 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxx6h\" (UniqueName: \"kubernetes.io/projected/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-kube-api-access-zxx6h\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605128 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-trusted-ca-bundle\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605145 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-serving-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605157 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-console-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.605292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.605169 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7aa370ef-944d-4957-9c7f-b44cb77cd0b0-oauth-serving-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:42:53.868080 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:53.868047 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zmhb7" Jul 22 12:42:54.056471 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.056433 2557 generic.go:356] "Generic (PLEG): container finished" podID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" containerID="8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a" exitCode=0 Jul 22 12:42:54.056665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.056497 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f67fbbcb-fq6wp" Jul 22 12:42:54.056665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.056531 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f67fbbcb-fq6wp" event={"ID":"7aa370ef-944d-4957-9c7f-b44cb77cd0b0","Type":"ContainerDied","Data":"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a"} Jul 22 12:42:54.056665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.056586 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f67fbbcb-fq6wp" event={"ID":"7aa370ef-944d-4957-9c7f-b44cb77cd0b0","Type":"ContainerDied","Data":"5c93d6d41599bb60e53e05251a5df1112b4147c27f7ed04f2a967353b27b65c6"} Jul 22 12:42:54.056665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.056607 2557 scope.go:117] "RemoveContainer" containerID="8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a" Jul 22 12:42:54.064857 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.064842 2557 scope.go:117] "RemoveContainer" containerID="8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a" Jul 22 12:42:54.065117 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:42:54.065097 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a\": container with ID starting with 8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a not found: ID does not exist" containerID="8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a" Jul 22 12:42:54.065178 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.065126 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a"} err="failed to get container status \"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a\": rpc error: code = NotFound desc = could not find container \"8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a\": container with ID starting with 8c1ac6ec3690eb8b6ab22ada49fb05270c4e4e8513260d633fd2945b74f3e82a not found: ID does not exist" Jul 22 12:42:54.085238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.085210 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:54.092481 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:54.092441 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f67fbbcb-fq6wp"] Jul 22 12:42:55.485099 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:42:55.485067 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" path="/var/lib/kubelet/pods/7aa370ef-944d-4957-9c7f-b44cb77cd0b0/volumes" Jul 22 12:43:14.504737 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:14.504670 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:14.568399 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:14.568363 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:19.884524 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:19.884469 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:19.903489 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:19.903459 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:20.176584 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:20.176508 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:29.353011 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:29.352975 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:29.453481 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:29.453454 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:33.643705 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.643663 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:33.644186 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644048 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="alertmanager" containerID="cri-o://c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" gracePeriod=120 Jul 22 12:43:33.644186 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644107 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-metric" containerID="cri-o://b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" gracePeriod=120 Jul 22 12:43:33.644186 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644151 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="prom-label-proxy" containerID="cri-o://f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" gracePeriod=120 Jul 22 12:43:33.644186 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644157 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy" containerID="cri-o://b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" gracePeriod=120 Jul 22 12:43:33.644415 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644183 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="config-reloader" containerID="cri-o://7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" gracePeriod=120 Jul 22 12:43:33.644415 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:33.644120 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-web" containerID="cri-o://9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" gracePeriod=120 Jul 22 12:43:34.214810 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214775 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" exitCode=0 Jul 22 12:43:34.214810 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214801 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" exitCode=0 Jul 22 12:43:34.214810 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214808 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" exitCode=0 Jul 22 12:43:34.214810 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214815 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" exitCode=0 Jul 22 12:43:34.215103 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214842 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc"} Jul 22 12:43:34.215103 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793"} Jul 22 12:43:34.215103 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214948 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d"} Jul 22 12:43:34.215103 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.214973 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213"} Jul 22 12:43:34.880520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.880496 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:34.957184 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957093 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957184 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957134 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957184 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957158 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957184 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957175 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957206 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957301 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957349 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957387 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957423 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957501 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957491 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhw9w\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957849 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957504 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:34.957849 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957521 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957849 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957561 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.957849 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957597 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web\") pod \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\" (UID: \"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d\") " Jul 22 12:43:34.958056 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957846 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:43:34.958056 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.957909 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-metrics-client-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:34.960815 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.960778 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w" (OuterVolumeSpecName: "kube-api-access-hhw9w") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "kube-api-access-hhw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:43:34.961203 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961043 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:43:34.961203 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961058 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.961203 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961128 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:34.961425 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961385 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume" (OuterVolumeSpecName: "config-volume") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.961568 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961530 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.961698 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.961682 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.963310 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.963287 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out" (OuterVolumeSpecName: "config-out") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:43:34.963390 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.963308 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.965250 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.965221 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:34.971073 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:34.971046 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config" (OuterVolumeSpecName: "web-config") pod "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" (UID: "0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:35.058728 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058669 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058728 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058703 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058742 2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-out\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058753 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhw9w\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-kube-api-access-hhw9w\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058762 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-main-tls\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058771 2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-tls-assets\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058779 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058788 2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-config-volume\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058796 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058805 2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-cluster-tls-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058813 2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-web-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.058896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.058821 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d-alertmanager-main-db\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:35.221834 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221749 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" exitCode=0 Jul 22 12:43:35.221834 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221774 2557 generic.go:356] "Generic (PLEG): container finished" podID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerID="9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" exitCode=0 Jul 22 12:43:35.221834 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221812 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a"} Jul 22 12:43:35.222052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221835 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.222052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221866 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136"} Jul 22 12:43:35.222052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221888 2557 scope.go:117] "RemoveContainer" containerID="f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" Jul 22 12:43:35.222052 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.221898 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d","Type":"ContainerDied","Data":"992a6a3b6daea72c88260561debe257e889d90435e943768aaa2fde6bd483bf8"} Jul 22 12:43:35.229772 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.229749 2557 scope.go:117] "RemoveContainer" containerID="b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" Jul 22 12:43:35.237212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.237190 2557 scope.go:117] "RemoveContainer" containerID="b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" Jul 22 12:43:35.244062 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.244042 2557 scope.go:117] "RemoveContainer" containerID="9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" Jul 22 12:43:35.247599 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.247577 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:35.251844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.251820 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:35.252010 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.251987 2557 scope.go:117] "RemoveContainer" containerID="7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" Jul 22 12:43:35.259024 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.259002 2557 scope.go:117] "RemoveContainer" containerID="c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" Jul 22 12:43:35.266207 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.266188 2557 scope.go:117] "RemoveContainer" containerID="f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60" Jul 22 12:43:35.273053 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.273035 2557 scope.go:117] "RemoveContainer" containerID="f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" Jul 22 12:43:35.273311 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.273293 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc\": container with ID starting with f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc not found: ID does not exist" containerID="f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" Jul 22 12:43:35.273368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.273319 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc"} err="failed to get container status \"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc\": rpc error: code = NotFound desc = could not find container \"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc\": container with ID starting with f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc not found: ID does not exist" Jul 22 12:43:35.273368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.273338 2557 scope.go:117] "RemoveContainer" containerID="b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" Jul 22 12:43:35.273574 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.273549 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a\": container with ID starting with b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a not found: ID does not exist" containerID="b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" Jul 22 12:43:35.273739 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.273575 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a"} err="failed to get container status \"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a\": rpc error: code = NotFound desc = could not find container \"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a\": container with ID starting with b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a not found: ID does not exist" Jul 22 12:43:35.273739 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.273589 2557 scope.go:117] "RemoveContainer" containerID="b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" Jul 22 12:43:35.274404 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.274384 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793\": container with ID starting with b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793 not found: ID does not exist" containerID="b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" Jul 22 12:43:35.274480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.274411 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793"} err="failed to get container status \"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793\": rpc error: code = NotFound desc = could not find container \"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793\": container with ID starting with b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793 not found: ID does not exist" Jul 22 12:43:35.274480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.274426 2557 scope.go:117] "RemoveContainer" containerID="9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" Jul 22 12:43:35.274734 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.274696 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136\": container with ID starting with 9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136 not found: ID does not exist" containerID="9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" Jul 22 12:43:35.274819 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.274739 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136"} err="failed to get container status \"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136\": rpc error: code = NotFound desc = could not find container \"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136\": container with ID starting with 9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136 not found: ID does not exist" Jul 22 12:43:35.274819 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.274755 2557 scope.go:117] "RemoveContainer" containerID="7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" Jul 22 12:43:35.275028 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.275013 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d\": container with ID starting with 7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d not found: ID does not exist" containerID="7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" Jul 22 12:43:35.275064 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275031 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d"} err="failed to get container status \"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d\": rpc error: code = NotFound desc = could not find container \"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d\": container with ID starting with 7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d not found: ID does not exist" Jul 22 12:43:35.275064 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275044 2557 scope.go:117] "RemoveContainer" containerID="c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" Jul 22 12:43:35.275265 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.275248 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213\": container with ID starting with c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213 not found: ID does not exist" containerID="c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" Jul 22 12:43:35.275299 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275271 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213"} err="failed to get container status \"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213\": rpc error: code = NotFound desc = could not find container \"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213\": container with ID starting with c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213 not found: ID does not exist" Jul 22 12:43:35.275299 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275289 2557 scope.go:117] "RemoveContainer" containerID="f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60" Jul 22 12:43:35.275498 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:35.275478 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60\": container with ID starting with f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60 not found: ID does not exist" containerID="f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60" Jul 22 12:43:35.275542 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275503 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60"} err="failed to get container status \"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60\": rpc error: code = NotFound desc = could not find container \"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60\": container with ID starting with f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60 not found: ID does not exist" Jul 22 12:43:35.275542 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275515 2557 scope.go:117] "RemoveContainer" containerID="f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc" Jul 22 12:43:35.275743 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275718 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc"} err="failed to get container status \"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc\": rpc error: code = NotFound desc = could not find container \"f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc\": container with ID starting with f1cc32e321aa69e52b607361d2127db9ee3846b8c9d383b4cb31bfca8e4598fc not found: ID does not exist" Jul 22 12:43:35.275743 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275743 2557 scope.go:117] "RemoveContainer" containerID="b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a" Jul 22 12:43:35.275962 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275945 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a"} err="failed to get container status \"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a\": rpc error: code = NotFound desc = could not find container \"b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a\": container with ID starting with b407e835cd2deb99eb3c776907b7ef3a0c616e95122e174b3dce5e9d24959d7a not found: ID does not exist" Jul 22 12:43:35.276036 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.275964 2557 scope.go:117] "RemoveContainer" containerID="b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793" Jul 22 12:43:35.276179 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276164 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793"} err="failed to get container status \"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793\": rpc error: code = NotFound desc = could not find container \"b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793\": container with ID starting with b5e9130d6639cbcc735c6a0f65996899bbff0843f26fcb2c48f95f309271a793 not found: ID does not exist" Jul 22 12:43:35.276233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276179 2557 scope.go:117] "RemoveContainer" containerID="9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136" Jul 22 12:43:35.276359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276343 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136"} err="failed to get container status \"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136\": rpc error: code = NotFound desc = could not find container \"9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136\": container with ID starting with 9de228d92ad4700d45ccaa44632084db20acc69cd8d1c686f7c6fa80511af136 not found: ID does not exist" Jul 22 12:43:35.276402 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276358 2557 scope.go:117] "RemoveContainer" containerID="7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d" Jul 22 12:43:35.276584 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276567 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d"} err="failed to get container status \"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d\": rpc error: code = NotFound desc = could not find container \"7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d\": container with ID starting with 7c1b8505be47122da85d104d3ad67490c94132daba148b99ee4342177071325d not found: ID does not exist" Jul 22 12:43:35.276621 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276588 2557 scope.go:117] "RemoveContainer" containerID="c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213" Jul 22 12:43:35.276776 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276760 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213"} err="failed to get container status \"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213\": rpc error: code = NotFound desc = could not find container \"c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213\": container with ID starting with c19be7892370ab674b00483011783c2792637050a2b6aae5e6619ee478826213 not found: ID does not exist" Jul 22 12:43:35.276828 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276776 2557 scope.go:117] "RemoveContainer" containerID="f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60" Jul 22 12:43:35.276994 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.276978 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60"} err="failed to get container status \"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60\": rpc error: code = NotFound desc = could not find container \"f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60\": container with ID starting with f1dbe714faa60cd02eb3e24caf0eff09eef41a3b45ec32230a8414f45db53a60 not found: ID does not exist" Jul 22 12:43:35.485478 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:35.485367 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" path="/var/lib/kubelet/pods/0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d/volumes" Jul 22 12:43:37.922350 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922295 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:37.922899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922747 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="prometheus" containerID="cri-o://443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" gracePeriod=600 Jul 22 12:43:37.922899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922787 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" gracePeriod=600 Jul 22 12:43:37.922899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922811 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-web" containerID="cri-o://daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" gracePeriod=600 Jul 22 12:43:37.922899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922811 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="thanos-sidecar" containerID="cri-o://5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" gracePeriod=600 Jul 22 12:43:37.922899 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.922826 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="config-reloader" containerID="cri-o://c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" gracePeriod=600 Jul 22 12:43:37.923188 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:37.923013 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy" containerID="cri-o://66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" gracePeriod=600 Jul 22 12:43:38.172097 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.172069 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.237085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237047 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" exitCode=0 Jul 22 12:43:38.237085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237078 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" exitCode=0 Jul 22 12:43:38.237085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237089 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" exitCode=0 Jul 22 12:43:38.237378 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237097 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" exitCode=0 Jul 22 12:43:38.237378 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237105 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" exitCode=0 Jul 22 12:43:38.237378 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237113 2557 generic.go:356] "Generic (PLEG): container finished" podID="6ce14185-505c-4333-bcee-46362a771187" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" exitCode=0 Jul 22 12:43:38.237378 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237152 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.237378 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237343 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237438 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237483 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237511 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237537 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237574 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} Jul 22 12:43:38.237642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237609 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} Jul 22 12:43:38.237958 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.237645 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ce14185-505c-4333-bcee-46362a771187","Type":"ContainerDied","Data":"0582cb7c59e1ae2667f1076c29f6eed15cc96a36bf1f31f105c282421907cae8"} Jul 22 12:43:38.245815 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.245793 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.252977 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.252956 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.260125 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.260106 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.267044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.267024 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.273653 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.273636 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.281405 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.281386 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.283541 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283518 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283563 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283599 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283690 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283622 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283690 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283657 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283690 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283680 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283724 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283801 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283841 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.283876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283870 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.284108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283904 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.284108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283942 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.284108 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.283984 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284387 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284447 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wxmt\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284505 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284537 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284565 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284597 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284621 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284656 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls\") pod \"6ce14185-505c-4333-bcee-46362a771187\" (UID: \"6ce14185-505c-4333-bcee-46362a771187\") " Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284925 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.285039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284945 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-metrics-client-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.286292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.284501 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.286292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.285195 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.286292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.286036 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.286292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.286185 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.286692 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.286579 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:38.287699 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.287229 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.287699 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.287264 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:43:38.287699 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.287394 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.287980 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.287926 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.288194 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.288120 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.288657 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.288583 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config" (OuterVolumeSpecName: "config") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.288657 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.288635 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.288875 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.288779 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:43:38.288935 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.288883 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.289232 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.289213 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.289969 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.289940 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out" (OuterVolumeSpecName: "config-out") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:43:38.290113 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.290099 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt" (OuterVolumeSpecName: "kube-api-access-5wxmt") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "kube-api-access-5wxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:43:38.291789 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.291772 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.292088 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.292066 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.292151 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292096 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.292151 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292114 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.292399 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.292382 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.292462 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292408 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.292462 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292436 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.292797 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.292775 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.292842 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292807 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.292842 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.292829 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.293107 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.293090 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.293168 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293113 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.293168 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293133 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.293410 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.293390 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.293448 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293419 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.293448 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293443 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.293695 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.293678 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.293776 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293703 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.293776 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.293743 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.293997 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:43:38.293978 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.294039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294003 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.294039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294022 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.294268 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294249 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.294330 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294268 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.294503 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294481 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.294541 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294506 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.294741 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294721 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.294790 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294743 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.294938 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294921 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.294984 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.294941 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.295139 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295121 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.295187 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295140 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.295368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295342 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.295368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295364 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.295587 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295566 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.295629 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295590 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.295854 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295832 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.295905 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.295855 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.296069 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296049 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.296069 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296070 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.296308 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296282 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.296387 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296308 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.296520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296504 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.296574 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296523 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.296786 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296759 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.296877 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.296787 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.297062 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297047 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.297111 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297064 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.297275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297258 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.297321 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297276 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.297363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297341 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config" (OuterVolumeSpecName: "web-config") pod "6ce14185-505c-4333-bcee-46362a771187" (UID: "6ce14185-505c-4333-bcee-46362a771187"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:38.297520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297501 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.297567 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297520 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.297740 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297721 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.297792 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297741 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.297973 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297957 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.298067 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.297975 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.298192 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298177 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.298192 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298192 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.298391 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298374 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.298430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298393 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.298592 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298576 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.298644 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298592 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.298790 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298774 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.298834 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298790 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.298956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298942 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.298992 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.298956 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.299125 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299109 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.299166 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299125 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.299311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299286 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.299311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299310 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.299520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299505 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.299557 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299521 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.299703 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299690 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.299766 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299703 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.299935 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299919 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.299978 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.299935 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.300165 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300148 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.300219 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300166 2557 scope.go:117] "RemoveContainer" containerID="1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310" Jul 22 12:43:38.300415 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300397 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310"} err="failed to get container status \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": rpc error: code = NotFound desc = could not find container \"1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310\": container with ID starting with 1f9dd6ebf26e2c4bfe08e8c5afafe28fbc07a0fddb5ff91edac2487673e0b310 not found: ID does not exist" Jul 22 12:43:38.300457 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300415 2557 scope.go:117] "RemoveContainer" containerID="66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766" Jul 22 12:43:38.300616 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300599 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766"} err="failed to get container status \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": rpc error: code = NotFound desc = could not find container \"66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766\": container with ID starting with 66b0cb468bb75a0c6839b0c60fbe2ff48b4f18e60abc5deb6c60bff645ff3766 not found: ID does not exist" Jul 22 12:43:38.300657 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300616 2557 scope.go:117] "RemoveContainer" containerID="daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163" Jul 22 12:43:38.300844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300828 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163"} err="failed to get container status \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": rpc error: code = NotFound desc = could not find container \"daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163\": container with ID starting with daef8a7b2b2a362906ff1ed4277d4e21af0f0043c810d2c384851730278e0163 not found: ID does not exist" Jul 22 12:43:38.300844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.300844 2557 scope.go:117] "RemoveContainer" containerID="5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b" Jul 22 12:43:38.301051 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301033 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b"} err="failed to get container status \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": rpc error: code = NotFound desc = could not find container \"5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b\": container with ID starting with 5e0107bc93318b2bc6751f60bcf92a6a6cffc1683130bfb9f06ffcaf86edb57b not found: ID does not exist" Jul 22 12:43:38.301089 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301055 2557 scope.go:117] "RemoveContainer" containerID="c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3" Jul 22 12:43:38.301254 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301238 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3"} err="failed to get container status \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": rpc error: code = NotFound desc = could not find container \"c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3\": container with ID starting with c9edab8de2264327b76bb6ecfb261d858b8161f8b4de12c5f94730df7c805ac3 not found: ID does not exist" Jul 22 12:43:38.301254 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301253 2557 scope.go:117] "RemoveContainer" containerID="443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b" Jul 22 12:43:38.301428 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301407 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b"} err="failed to get container status \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": rpc error: code = NotFound desc = could not find container \"443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b\": container with ID starting with 443443ad5dacce1fe85f7ecffe0927d956f3ede4115110c3ff9cd1e9eb09a99b not found: ID does not exist" Jul 22 12:43:38.301493 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301431 2557 scope.go:117] "RemoveContainer" containerID="eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c" Jul 22 12:43:38.301622 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.301607 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c"} err="failed to get container status \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": rpc error: code = NotFound desc = could not find container \"eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c\": container with ID starting with eeb2ea00e1549e4cdd6e3af4d23f42f995cbe272c6785cc54f0501abd2d3f94c not found: ID does not exist" Jul 22 12:43:38.385267 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385236 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxmt\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-kube-api-access-5wxmt\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385267 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385262 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385267 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385273 2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385285 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385294 2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-config-out\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385302 2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ce14185-505c-4333-bcee-46362a771187-tls-assets\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385310 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385319 2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385327 2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-kube-rbac-proxy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385335 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385344 2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-metrics-client-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385354 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385361 2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-web-config\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385369 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385377 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385385 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385395 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ce14185-505c-4333-bcee-46362a771187-prometheus-k8s-db\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.385465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.385403 2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ce14185-505c-4333-bcee-46362a771187-secret-grpc-tls\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:43:38.566338 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.566301 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:38.572970 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.572944 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:38.623783 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.623752 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:38.624014 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624000 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="prometheus" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624017 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="prometheus" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624024 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-thanos" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624030 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-thanos" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624037 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="config-reloader" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624042 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="config-reloader" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624051 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="alertmanager" Jul 22 12:43:38.624065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624061 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="alertmanager" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624069 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" containerName="console" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624075 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" containerName="console" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624082 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="prom-label-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624087 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="prom-label-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624095 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" containerName="console" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624100 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" containerName="console" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624106 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624111 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624119 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624127 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624140 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624145 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624151 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="thanos-sidecar" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624156 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="thanos-sidecar" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624163 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="init-config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624168 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="init-config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624173 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="init-config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624177 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="init-config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624183 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624187 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624192 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-metric" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624197 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-metric" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624202 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624210 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624253 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624261 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="config-reloader" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624267 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-metric" Jul 22 12:43:38.624261 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624272 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624276 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624281 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9e0b17-5bc0-45ec-b1ba-b189bd4f4ec1" containerName="console" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624290 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa370ef-944d-4957-9c7f-b44cb77cd0b0" containerName="console" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624297 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="alertmanager" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624313 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy-web" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624318 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="prom-label-proxy" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624323 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="prometheus" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624327 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09bbd3-d78b-45a4-a508-b7a18f2d9e8d" containerName="kube-rbac-proxy" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624331 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="thanos-sidecar" Jul 22 12:43:38.625028 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.624335 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce14185-505c-4333-bcee-46362a771187" containerName="kube-rbac-proxy-thanos" Jul 22 12:43:38.627656 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.627637 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.630757 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.630738 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 12:43:38.630986 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.630975 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 12:43:38.631154 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.631133 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 12:43:38.631324 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.631310 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 12:43:38.632676 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.632662 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dhujsck446n6c\"" Jul 22 12:43:38.633401 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.633380 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 12:43:38.633753 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.633734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 12:43:38.635273 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635232 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 12:43:38.635273 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635247 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 12:43:38.635505 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635286 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 12:43:38.635655 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635637 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 12:43:38.635743 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635722 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 12:43:38.635858 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 12:43:38.635949 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.635864 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 12:43:38.636344 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.636326 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nq5x\"" Jul 22 12:43:38.638655 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.638638 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 12:43:38.642662 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.642643 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 12:43:38.655624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.655595 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:38.687526 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687492 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687526 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687600 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687734 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.687793 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687802 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687819 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2ds\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-kube-api-access-vx2ds\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687902 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687919 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688034 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.687998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.688426 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.688037 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.788972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.788920 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.788972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.788974 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789031 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789125 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789192 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789231 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789215 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789239 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789258 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2ds\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-kube-api-access-vx2ds\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789283 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789334 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.789600 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.789464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.791904 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.791769 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.792150 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.792131 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.792243 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.792189 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794234 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.793274 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794234 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.793480 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794234 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.793550 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794234 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.794071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794762 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.794685 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.794870 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.794968 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.794950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.795060 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.795038 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.795285 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.795260 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.795359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.795315 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-config\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.795511 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.795490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6444703-53ba-42ad-8871-5f01350b043d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.796054 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.796032 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6444703-53ba-42ad-8871-5f01350b043d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.797064 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.797040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.799507 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.799480 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.799605 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.799553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.800582 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.800552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6444703-53ba-42ad-8871-5f01350b043d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.805860 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.805838 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2ds\" (UniqueName: \"kubernetes.io/projected/f6444703-53ba-42ad-8871-5f01350b043d-kube-api-access-vx2ds\") pod \"prometheus-k8s-0\" (UID: \"f6444703-53ba-42ad-8871-5f01350b043d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:38.936639 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:38.936607 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:39.090948 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:39.090922 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 12:43:39.093447 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:43:39.093412 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6444703_53ba_42ad_8871_5f01350b043d.slice/crio-58317d75c298c8cd082a4247cc57bb172f12e5b9725a6fc4569a6919bb75ba96 WatchSource:0}: Error finding container 58317d75c298c8cd082a4247cc57bb172f12e5b9725a6fc4569a6919bb75ba96: Status 404 returned error can't find the container with id 58317d75c298c8cd082a4247cc57bb172f12e5b9725a6fc4569a6919bb75ba96 Jul 22 12:43:39.242576 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:39.242537 2557 generic.go:356] "Generic (PLEG): container finished" podID="f6444703-53ba-42ad-8871-5f01350b043d" containerID="1e14bae4afaf5b36f9acb0e3b013ee59ad293a0303c9976984a3b92210117b22" exitCode=0 Jul 22 12:43:39.242778 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:39.242631 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerDied","Data":"1e14bae4afaf5b36f9acb0e3b013ee59ad293a0303c9976984a3b92210117b22"} Jul 22 12:43:39.242778 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:39.242739 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"58317d75c298c8cd082a4247cc57bb172f12e5b9725a6fc4569a6919bb75ba96"} Jul 22 12:43:39.486730 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:39.486669 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce14185-505c-4333-bcee-46362a771187" path="/var/lib/kubelet/pods/6ce14185-505c-4333-bcee-46362a771187/volumes" Jul 22 12:43:40.251110 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"22141f31cc6efaa4df218e6078085128e0d73d3b16e34876605eb1ae755b3f81"} Jul 22 12:43:40.251467 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251143 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"73a9ed32f7107383649ceb438856ce0b8a8c9a12e3d9b53ed89fd7d830938f0d"} Jul 22 12:43:40.251467 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"26dc56fa5088a2a13285dea9b7262a06c374ad4c5f2beb0de372a1cde10d911b"} Jul 22 12:43:40.251467 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251198 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"34fa41054f2d2f4fc5bd172613ca7ad1ec46cd61c1624a31778926887146b721"} Jul 22 12:43:40.251467 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251225 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"a89d61aaeedc1bd59a8e4b781031e7c43b4b3b0931346b249382ad3123c50b69"} Jul 22 12:43:40.251467 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.251248 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6444703-53ba-42ad-8871-5f01350b043d","Type":"ContainerStarted","Data":"420230bd37f5d5122b40015b1b4fd10b25607d84d996f5a6bbd10cc7edb7980a"} Jul 22 12:43:40.284696 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:40.284630 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.26479117 podStartE2EDuration="2.28461177s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.8206ms" isStatefulPod=false podCreationTimestamp="2026-07-22 12:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:43:40.282335109 +0000 UTC m=+159.369824290" watchObservedRunningTime="2026-07-22 12:43:40.28461177 +0000 UTC m=+159.372100969" Jul 22 12:43:41.222869 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:41.222834 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:41.303410 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:41.303369 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:43.937642 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:43.937602 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:43:50.610823 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:50.610789 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:50.668265 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:50.668229 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:53.260143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:53.260065 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:57.193906 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:57.193876 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:57.294756 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:57.294724 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:59.342372 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:59.342341 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:43:59.431280 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:43:59.431252 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:01.971747 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:01.971696 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:02.032751 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:02.032697 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:02.097538 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:02.097502 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:20.288926 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:20.288895 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:20.394079 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:20.394040 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:38.937773 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:38.937705 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:44:38.954074 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:38.954046 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:44:39.483616 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:39.483592 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 12:44:42.059192 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:42.059157 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:42.154685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:42.154643 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:42.247134 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:42.247097 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:44:49.941625 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:44:49.941588 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:12.609638 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:12.609595 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:12.669720 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:12.669668 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:34.443374 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:34.443281 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:35.305282 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:35.305251 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:38.255789 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:38.255757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:38.313137 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:38.313107 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:39.655430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:39.655391 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:40.093995 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:40.093964 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:40.412680 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:40.412602 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:40.528359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:40.528324 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:41.869209 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:41.869178 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:42.071499 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:42.071470 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:45.052074 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:45.052043 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:45.114143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:45.114114 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:48.467663 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:48.467625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:48.593659 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:48.593623 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:50.486883 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:50.486846 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:51.478507 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:51.478471 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:51.893138 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:51.893049 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:53.574109 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.574069 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh"] Jul 22 12:45:53.575911 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.575895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.579449 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.579426 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 12:45:53.580233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.580213 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-65f95\"" Jul 22 12:45:53.580341 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.580246 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:45:53.590348 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.590319 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh"] Jul 22 12:45:53.630885 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.630836 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb87q\" (UniqueName: \"kubernetes.io/projected/10654824-7f2a-4b90-87a2-7f91b422a3d1-kube-api-access-nb87q\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.631082 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.630893 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10654824-7f2a-4b90-87a2-7f91b422a3d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.732050 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.732007 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb87q\" (UniqueName: \"kubernetes.io/projected/10654824-7f2a-4b90-87a2-7f91b422a3d1-kube-api-access-nb87q\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.732232 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.732062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10654824-7f2a-4b90-87a2-7f91b422a3d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.732454 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.732437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10654824-7f2a-4b90-87a2-7f91b422a3d1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.742509 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.742475 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb87q\" (UniqueName: \"kubernetes.io/projected/10654824-7f2a-4b90-87a2-7f91b422a3d1-kube-api-access-nb87q\") pod \"openshift-lws-operator-fd8ccff4c-jklkh\" (UID: \"10654824-7f2a-4b90-87a2-7f91b422a3d1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:53.892552 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:53.892461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" Jul 22 12:45:54.036110 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:54.036080 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh"] Jul 22 12:45:54.039330 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:45:54.039297 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10654824_7f2a_4b90_87a2_7f91b422a3d1.slice/crio-905171a9f61d14e9dda64d28dbe8d2de760aaa8544170b7c0ad9dfb6700a8ea8 WatchSource:0}: Error finding container 905171a9f61d14e9dda64d28dbe8d2de760aaa8544170b7c0ad9dfb6700a8ea8: Status 404 returned error can't find the container with id 905171a9f61d14e9dda64d28dbe8d2de760aaa8544170b7c0ad9dfb6700a8ea8 Jul 22 12:45:54.718978 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:54.718945 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" event={"ID":"10654824-7f2a-4b90-87a2-7f91b422a3d1","Type":"ContainerStarted","Data":"905171a9f61d14e9dda64d28dbe8d2de760aaa8544170b7c0ad9dfb6700a8ea8"} Jul 22 12:45:57.731999 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:57.731906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" event={"ID":"10654824-7f2a-4b90-87a2-7f91b422a3d1","Type":"ContainerStarted","Data":"5e72ce135adc16c47afe84fb1b2d10e25657deef54ca432b5b641b5fb8927892"} Jul 22 12:45:58.680598 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:58.680563 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:45:59.995520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:45:59.995486 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:00.743038 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:00.743000 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:01.347229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:01.347203 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 12:46:16.109856 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:16.109797 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:16.211291 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:16.211257 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:16.961509 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:16.961457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:17.420528 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.420424 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:17.517379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.517341 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:17.560379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.560314 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jklkh" podStartSLOduration=21.148056799 podStartE2EDuration="24.560296941s" totalImagesPullingTime="3.412240142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:45:57.765304388 +0000 UTC m=+296.852793570" watchObservedRunningTime="2026-07-22 12:46:17.560296941 +0000 UTC m=+316.647786122" Jul 22 12:46:17.561647 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.561618 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8"] Jul 22 12:46:17.564913 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.564885 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.567685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.567653 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 12:46:17.567685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.567681 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 12:46:17.567910 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.567727 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 12:46:17.568035 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.568020 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nqhtb\"" Jul 22 12:46:17.568103 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.568057 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 12:46:17.586816 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.586775 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8"] Jul 22 12:46:17.629425 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.629379 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.629425 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.629426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.629663 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.629467 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.629663 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.629548 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvwk\" (UniqueName: \"kubernetes.io/projected/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-kube-api-access-kzvwk\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.629663 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.629603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.726240 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.726197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:17.730284 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.730380 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.730441 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.730516 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvwk\" (UniqueName: \"kubernetes.io/projected/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-kube-api-access-kzvwk\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.730572 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.730994 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.730965 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.732820 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.732789 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.732960 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.732859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.732960 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.732941 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.739884 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.739853 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvwk\" (UniqueName: \"kubernetes.io/projected/0e24bdfe-a9c0-4ecd-a07d-c658438ee151-kube-api-access-kzvwk\") pod \"opendatahub-operator-controller-manager-64bbff868f-w8dv8\" (UID: \"0e24bdfe-a9c0-4ecd-a07d-c658438ee151\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.828382 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.828335 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:17.876555 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.876518 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:17.925502 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:17.925466 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:18.036967 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.036873 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8"] Jul 22 12:46:18.040672 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.040634 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:18.042091 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:46:18.042056 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e24bdfe_a9c0_4ecd_a07d_c658438ee151.slice/crio-dd69661550d5908fa91d494f14a0f4dc7eb39dd454377cd1e2371f969191b313 WatchSource:0}: Error finding container dd69661550d5908fa91d494f14a0f4dc7eb39dd454377cd1e2371f969191b313: Status 404 returned error can't find the container with id dd69661550d5908fa91d494f14a0f4dc7eb39dd454377cd1e2371f969191b313 Jul 22 12:46:18.044274 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.044242 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:46:18.135058 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.135018 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:18.237308 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.237277 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:18.813136 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:18.813093 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" event={"ID":"0e24bdfe-a9c0-4ecd-a07d-c658438ee151","Type":"ContainerStarted","Data":"dd69661550d5908fa91d494f14a0f4dc7eb39dd454377cd1e2371f969191b313"} Jul 22 12:46:19.114039 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:19.113915 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:19.204546 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:19.204507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:21.827553 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:21.827516 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" event={"ID":"0e24bdfe-a9c0-4ecd-a07d-c658438ee151","Type":"ContainerStarted","Data":"b063a04db2090e0ab938797a2d495dd0b236c4b496b3f28d7f557c25db7030e5"} Jul 22 12:46:22.832637 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:22.832601 2557 generic.go:356] "Generic (PLEG): container finished" podID="0e24bdfe-a9c0-4ecd-a07d-c658438ee151" containerID="b063a04db2090e0ab938797a2d495dd0b236c4b496b3f28d7f557c25db7030e5" exitCode=0 Jul 22 12:46:22.833156 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:22.832668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" event={"ID":"0e24bdfe-a9c0-4ecd-a07d-c658438ee151","Type":"ContainerDied","Data":"b063a04db2090e0ab938797a2d495dd0b236c4b496b3f28d7f557c25db7030e5"} Jul 22 12:46:23.838065 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:23.838028 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" event={"ID":"0e24bdfe-a9c0-4ecd-a07d-c658438ee151","Type":"ContainerStarted","Data":"4d8f29d5ff2d36c0ae55abec3cb3c38b86532d4ee6c7e0a188e11ec496195b4e"} Jul 22 12:46:23.838513 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:23.838171 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:23.864254 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:23.864138 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" podStartSLOduration=2.474887646 podStartE2EDuration="6.864118421s" totalImagesPullingTime="3.304398444s" totalInitContainerRuntime="1.084832331s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:23.862405959 +0000 UTC m=+322.949895129" watchObservedRunningTime="2026-07-22 12:46:23.864118421 +0000 UTC m=+322.951607602" Jul 22 12:46:33.606401 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:33.606363 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:33.687204 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:33.687168 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:34.845425 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:34.845398 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-w8dv8" Jul 22 12:46:36.632018 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:36.631988 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:36.833430 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:36.833396 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.150863 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.150824 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.218543 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.218506 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9"] Jul 22 12:46:37.226236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.226217 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.229275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.229253 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 12:46:37.229544 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.229529 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 12:46:37.230384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.230368 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 12:46:37.230437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.230390 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 12:46:37.234772 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.234750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9"] Jul 22 12:46:37.328601 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.328564 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.395577 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.395540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hb8q\" (UniqueName: \"kubernetes.io/projected/b4ee986d-c002-40f9-afed-d233754a238e-kube-api-access-7hb8q\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.395751 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.395602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ee986d-c002-40f9-afed-d233754a238e-tmp\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.395751 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.395660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ee986d-c002-40f9-afed-d233754a238e-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.496423 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.496385 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ee986d-c002-40f9-afed-d233754a238e-tmp\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.496616 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.496477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ee986d-c002-40f9-afed-d233754a238e-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.496616 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.496551 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hb8q\" (UniqueName: \"kubernetes.io/projected/b4ee986d-c002-40f9-afed-d233754a238e-kube-api-access-7hb8q\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.498096 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.498075 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j9tp8"] Jul 22 12:46:37.498827 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.498809 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ee986d-c002-40f9-afed-d233754a238e-tmp\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.498925 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.498871 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ee986d-c002-40f9-afed-d233754a238e-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.501148 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.501117 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.504521 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.504496 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 12:46:37.504625 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.504497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jqgdd\"" Jul 22 12:46:37.517265 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.517240 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hb8q\" (UniqueName: \"kubernetes.io/projected/b4ee986d-c002-40f9-afed-d233754a238e-kube-api-access-7hb8q\") pod \"kube-auth-proxy-76c9b95cb4-js4b9\" (UID: \"b4ee986d-c002-40f9-afed-d233754a238e\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.519190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.519172 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j9tp8"] Jul 22 12:46:37.536830 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.536807 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" Jul 22 12:46:37.575257 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.575215 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.597686 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.597656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkrl\" (UniqueName: \"kubernetes.io/projected/ad574278-7887-4d95-a7ba-06b85ca52bb3-kube-api-access-5kkrl\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.597874 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.597774 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad574278-7887-4d95-a7ba-06b85ca52bb3-cert\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.681159 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.681122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9"] Jul 22 12:46:37.695075 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:46:37.695050 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ee986d_c002_40f9_afed_d233754a238e.slice/crio-080a1455dc5631c5d9265c6cd16aa3f78f601f24b46639039f735af45820b6be WatchSource:0}: Error finding container 080a1455dc5631c5d9265c6cd16aa3f78f601f24b46639039f735af45820b6be: Status 404 returned error can't find the container with id 080a1455dc5631c5d9265c6cd16aa3f78f601f24b46639039f735af45820b6be Jul 22 12:46:37.698586 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.698566 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad574278-7887-4d95-a7ba-06b85ca52bb3-cert\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.698691 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.698626 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkrl\" (UniqueName: \"kubernetes.io/projected/ad574278-7887-4d95-a7ba-06b85ca52bb3-kube-api-access-5kkrl\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.701033 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.701009 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad574278-7887-4d95-a7ba-06b85ca52bb3-cert\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.718919 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.718896 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkrl\" (UniqueName: \"kubernetes.io/projected/ad574278-7887-4d95-a7ba-06b85ca52bb3-kube-api-access-5kkrl\") pod \"odh-model-controller-7998c67dc4-j9tp8\" (UID: \"ad574278-7887-4d95-a7ba-06b85ca52bb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.771201 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.771114 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.819287 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.819256 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:37.870008 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.869971 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:37.891542 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.891513 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" event={"ID":"b4ee986d-c002-40f9-afed-d233754a238e","Type":"ContainerStarted","Data":"080a1455dc5631c5d9265c6cd16aa3f78f601f24b46639039f735af45820b6be"} Jul 22 12:46:37.947102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:37.947079 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j9tp8"] Jul 22 12:46:37.948897 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:46:37.948871 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad574278_7887_4d95_a7ba_06b85ca52bb3.slice/crio-55dfc52269872d8cdaec673633d119ded0135d0f7607b9357be217c4b7507a9b WatchSource:0}: Error finding container 55dfc52269872d8cdaec673633d119ded0135d0f7607b9357be217c4b7507a9b: Status 404 returned error can't find the container with id 55dfc52269872d8cdaec673633d119ded0135d0f7607b9357be217c4b7507a9b Jul 22 12:46:38.724543 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:38.724501 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:38.899696 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:38.899661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" event={"ID":"ad574278-7887-4d95-a7ba-06b85ca52bb3","Type":"ContainerStarted","Data":"55dfc52269872d8cdaec673633d119ded0135d0f7607b9357be217c4b7507a9b"} Jul 22 12:46:38.980414 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:38.980301 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:39.230612 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:39.230529 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:39.435366 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:39.435336 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:39.585524 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:39.585434 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:39.695421 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:39.695384 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:41.636770 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.636672 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:41.913954 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.913868 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" event={"ID":"b4ee986d-c002-40f9-afed-d233754a238e","Type":"ContainerStarted","Data":"6580f72af93cdd0fe9d367dccb77bf14f743865dd11b5db89cf51453e5156e9f"} Jul 22 12:46:41.915336 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.915302 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" event={"ID":"ad574278-7887-4d95-a7ba-06b85ca52bb3","Type":"ContainerStarted","Data":"76d32303a16644f6ed9c70c88a20de513b822b32538d1aa6699dd9f434f06c74"} Jul 22 12:46:41.915465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.915389 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:41.938232 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.938181 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-js4b9" podStartSLOduration=1.3981578479999999 podStartE2EDuration="4.938169038s" totalImagesPullingTime="3.54001119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:41.936958335 +0000 UTC m=+341.024447515" watchObservedRunningTime="2026-07-22 12:46:41.938169038 +0000 UTC m=+341.025658217" Jul 22 12:46:41.959027 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:41.958982 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" podStartSLOduration=1.626345516 podStartE2EDuration="4.958965631s" totalImagesPullingTime="3.332620115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:41.958499561 +0000 UTC m=+341.045988740" watchObservedRunningTime="2026-07-22 12:46:41.958965631 +0000 UTC m=+341.046454803" Jul 22 12:46:43.664975 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:43.664939 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:44.077145 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.077113 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:44.820159 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.820124 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7"] Jul 22 12:46:44.823507 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.823482 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.826784 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.826765 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q8j98\"" Jul 22 12:46:44.827105 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.827089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 12:46:44.840587 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.840564 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7"] Jul 22 12:46:44.860082 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860052 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860088 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860156 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860192 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbzg\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-kube-api-access-wrbzg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860270 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.860437 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.860357 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a166b8e-6365-4bc8-8c4e-37895d574c38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.897297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.897261 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:44.961190 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961154 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961231 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a166b8e-6365-4bc8-8c4e-37895d574c38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961261 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961362 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961337 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961381 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbzg\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-kube-api-access-wrbzg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961860 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961662 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961860 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961844 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.961946 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.961914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.962049 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.962031 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a166b8e-6365-4bc8-8c4e-37895d574c38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.963633 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.963614 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.963732 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.963692 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:44.983805 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:44.983776 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:45.003928 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.003901 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbzg\" (UniqueName: \"kubernetes.io/projected/9a166b8e-6365-4bc8-8c4e-37895d574c38-kube-api-access-wrbzg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4dsbl7\" (UID: \"9a166b8e-6365-4bc8-8c4e-37895d574c38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:45.133152 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.133055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:45.283913 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.283870 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:45.293803 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.293614 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7"] Jul 22 12:46:45.296229 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:46:45.296202 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a166b8e_6365_4bc8_8c4e_37895d574c38.slice/crio-1b4ed0db1971518f4b42f3eedc292657c454df462cf33932b122476209c6e8f3 WatchSource:0}: Error finding container 1b4ed0db1971518f4b42f3eedc292657c454df462cf33932b122476209c6e8f3: Status 404 returned error can't find the container with id 1b4ed0db1971518f4b42f3eedc292657c454df462cf33932b122476209c6e8f3 Jul 22 12:46:45.402384 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.402306 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:45.615326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.615291 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh"] Jul 22 12:46:45.619665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.619648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.623176 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.623160 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 12:46:45.623761 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.623745 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gvbqk\"" Jul 22 12:46:45.623816 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.623747 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 12:46:45.624198 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.624178 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 12:46:45.638213 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.638188 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh"] Jul 22 12:46:45.665379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.665305 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3befb592-c4a5-48f6-8194-9ce298b5ccd8-manager-config\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.665379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.665344 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw929\" (UniqueName: \"kubernetes.io/projected/3befb592-c4a5-48f6-8194-9ce298b5ccd8-kube-api-access-nw929\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.665379 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.665374 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.665585 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.665415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-metrics-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.766402 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.766366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3befb592-c4a5-48f6-8194-9ce298b5ccd8-manager-config\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.766402 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.766417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw929\" (UniqueName: \"kubernetes.io/projected/3befb592-c4a5-48f6-8194-9ce298b5ccd8-kube-api-access-nw929\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.766677 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.766455 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.766677 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.766479 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-metrics-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.767096 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.767073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3befb592-c4a5-48f6-8194-9ce298b5ccd8-manager-config\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.769074 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.769048 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.769154 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.769056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3befb592-c4a5-48f6-8194-9ce298b5ccd8-metrics-cert\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.781923 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.781899 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw929\" (UniqueName: \"kubernetes.io/projected/3befb592-c4a5-48f6-8194-9ce298b5ccd8-kube-api-access-nw929\") pod \"lws-controller-manager-8b6495d78-2lvwh\" (UID: \"3befb592-c4a5-48f6-8194-9ce298b5ccd8\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.928372 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.928244 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:45.931984 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:45.931958 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" event={"ID":"9a166b8e-6365-4bc8-8c4e-37895d574c38","Type":"ContainerStarted","Data":"1b4ed0db1971518f4b42f3eedc292657c454df462cf33932b122476209c6e8f3"} Jul 22 12:46:46.069449 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:46.069415 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh"] Jul 22 12:46:46.073933 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:46:46.073900 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3befb592_c4a5_48f6_8194_9ce298b5ccd8.slice/crio-f9d7833d7ae5d79da0fa5ddb60e5ffcb112927e4a8eb8cbaa773ef0a0e6586ab WatchSource:0}: Error finding container f9d7833d7ae5d79da0fa5ddb60e5ffcb112927e4a8eb8cbaa773ef0a0e6586ab: Status 404 returned error can't find the container with id f9d7833d7ae5d79da0fa5ddb60e5ffcb112927e4a8eb8cbaa773ef0a0e6586ab Jul 22 12:46:46.938844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:46.938796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" event={"ID":"3befb592-c4a5-48f6-8194-9ce298b5ccd8","Type":"ContainerStarted","Data":"f9d7833d7ae5d79da0fa5ddb60e5ffcb112927e4a8eb8cbaa773ef0a0e6586ab"} Jul 22 12:46:48.494281 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.494242 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:46:48.494693 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.494334 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:46:48.494693 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.494384 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:46:48.949564 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.949530 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" event={"ID":"3befb592-c4a5-48f6-8194-9ce298b5ccd8","Type":"ContainerStarted","Data":"095f80c105f1350a5010fc03472a9f4097195a5c342948d0e8225ddd5f732ae0"} Jul 22 12:46:48.949764 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.949590 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:46:48.951207 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.951172 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" event={"ID":"9a166b8e-6365-4bc8-8c4e-37895d574c38","Type":"ContainerStarted","Data":"c5f5cd93179e0ca84995adcb636088c68d943614de1abbeb71080ca35bb48427"} Jul 22 12:46:48.987564 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:48.987510 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" podStartSLOduration=1.5701433439999999 podStartE2EDuration="3.987495027s" totalImagesPullingTime="2.417351683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:48.984189589 +0000 UTC m=+348.071678768" watchObservedRunningTime="2026-07-22 12:46:48.987495027 +0000 UTC m=+348.074984206" Jul 22 12:46:49.093684 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:49.093635 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" podStartSLOduration=1.899810271 podStartE2EDuration="5.093618886s" totalImagesPullingTime="3.193808615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:49.091492015 +0000 UTC m=+348.178981198" watchObservedRunningTime="2026-07-22 12:46:49.093618886 +0000 UTC m=+348.181108068" Jul 22 12:46:49.134178 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:49.134143 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:49.138482 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:49.138457 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:49.954765 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:49.954725 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:49.955807 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:49.955787 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4dsbl7" Jul 22 12:46:50.383675 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:50.383575 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:50.592374 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:50.592340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:50.785791 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:50.785757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:50.887465 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:50.887427 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:51.015696 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:51.015661 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:51.339077 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:51.339043 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:46:52.921475 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:52.921391 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-j9tp8" Jul 22 12:46:59.956801 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:46:59.956770 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-2lvwh" Jul 22 12:47:03.571143 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:03.571100 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:03.649355 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:03.649310 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:36.169896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:36.169854 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:36.270864 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:36.270830 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:38.227937 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:38.227901 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:38.334945 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:38.334905 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:38.502452 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:38.502366 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:39.196607 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:39.196568 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:39.449911 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:39.449827 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:41.006435 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:41.006401 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:41.098554 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:41.098516 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:46.895896 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:46.895857 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:46.996782 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:46.996751 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:47:47.098485 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:47:47.098457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:02.509292 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:02.509256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:05.864908 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:05.864871 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:06.706152 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:06.706125 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:10.286245 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:10.286210 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:10.582558 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:10.582470 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:10.695621 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:10.695577 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:12.695897 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:12.695857 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:13.090576 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:13.090494 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:13.618311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:13.618275 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:15.579856 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:15.579822 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:18.499556 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:18.499514 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:18.617648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:18.617614 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:23.884634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:23.884549 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:23.982940 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:23.982914 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:24.296611 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.296579 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:24.413412 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.413370 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:24.910802 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.910770 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7"] Jul 22 12:48:24.913890 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.913871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:24.916411 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.916390 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 12:48:24.917326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.917310 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 12:48:24.917402 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.917313 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g556c\"" Jul 22 12:48:24.917402 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.917365 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 12:48:24.917879 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.917861 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 12:48:24.926159 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.926139 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7"] Jul 22 12:48:24.944000 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.943976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b205d99-3af7-4f56-930f-fe6f768d146e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:24.944102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.944020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b205d99-3af7-4f56-930f-fe6f768d146e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:24.944102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.944049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztm76\" (UniqueName: \"kubernetes.io/projected/1b205d99-3af7-4f56-930f-fe6f768d146e-kube-api-access-ztm76\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:24.960133 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:24.960102 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:25.042130 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.042103 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:25.045193 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.045169 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b205d99-3af7-4f56-930f-fe6f768d146e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.045310 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.045230 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b205d99-3af7-4f56-930f-fe6f768d146e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.045310 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.045281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztm76\" (UniqueName: \"kubernetes.io/projected/1b205d99-3af7-4f56-930f-fe6f768d146e-kube-api-access-ztm76\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.045825 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.045808 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b205d99-3af7-4f56-930f-fe6f768d146e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.047588 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.047567 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b205d99-3af7-4f56-930f-fe6f768d146e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.053725 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.053678 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztm76\" (UniqueName: \"kubernetes.io/projected/1b205d99-3af7-4f56-930f-fe6f768d146e-kube-api-access-ztm76\") pod \"kuadrant-console-plugin-66b5774985-qvqx7\" (UID: \"1b205d99-3af7-4f56-930f-fe6f768d146e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.222749 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.222695 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" Jul 22 12:48:25.352624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:25.352592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7"] Jul 22 12:48:25.355574 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:48:25.355537 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b205d99_3af7_4f56_930f_fe6f768d146e.slice/crio-04084b8d1e11a6f327958c2810afa2585058229fdb2266dcb014bcfce5cc7a1f WatchSource:0}: Error finding container 04084b8d1e11a6f327958c2810afa2585058229fdb2266dcb014bcfce5cc7a1f: Status 404 returned error can't find the container with id 04084b8d1e11a6f327958c2810afa2585058229fdb2266dcb014bcfce5cc7a1f Jul 22 12:48:26.341377 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:26.341338 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" event={"ID":"1b205d99-3af7-4f56-930f-fe6f768d146e","Type":"ContainerStarted","Data":"04084b8d1e11a6f327958c2810afa2585058229fdb2266dcb014bcfce5cc7a1f"} Jul 22 12:48:27.725318 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:27.725123 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:27.825520 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:27.825490 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:30.289665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:30.289616 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:30.392972 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:30.392893 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:32.830665 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:32.830596 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:37.091807 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:37.091759 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:37.262438 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:37.262377 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:42.361294 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:42.361233 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:42.462301 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:42.462256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:42.772550 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:42.772512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:42.874923 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:42.874878 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:46.147027 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:46.146992 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:46.251579 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:46.251537 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:47.452049 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:47.452015 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" event={"ID":"1b205d99-3af7-4f56-930f-fe6f768d146e","Type":"ContainerStarted","Data":"505d864cddbd0b6fdee2c3b333ce8a3434721be133631849f1204224d89616bd"} Jul 22 12:48:47.491608 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:47.491542 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qvqx7" podStartSLOduration=1.832701874 podStartE2EDuration="23.491521107s" totalImagesPullingTime="21.658819233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:47.490611459 +0000 UTC m=+466.578100653" watchObservedRunningTime="2026-07-22 12:48:47.491521107 +0000 UTC m=+466.579010289" Jul 22 12:48:48.792485 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:48.792451 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:48.894658 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:48.894622 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:50.517420 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:50.517389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:48:50.618743 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:48:50.618691 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:09.523049 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.523007 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg"] Jul 22 12:49:09.554014 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.553981 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:09.660604 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.660565 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg"] Jul 22 12:49:09.660838 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.660733 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.663275 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.663252 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-727dl\"" Jul 22 12:49:09.747973 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.747929 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.747982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748013 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748163 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748050 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6ea09-f03e-4b79-8f3b-b26007b967de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748348 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748163 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748468 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748539 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748524 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748597 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.748648 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.748606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltcn\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-kube-api-access-dltcn\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.849927 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849840 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.849927 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.849927 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltcn\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-kube-api-access-dltcn\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849935 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.849993 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6ea09-f03e-4b79-8f3b-b26007b967de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850212 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850113 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850537 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850296 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850537 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850393 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850647 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850563 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.850647 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850626 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.851011 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.850993 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a6ea09-f03e-4b79-8f3b-b26007b967de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.852294 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.852272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.852698 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.852678 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.859343 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.859307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltcn\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-kube-api-access-dltcn\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.859459 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.859348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a6ea09-f03e-4b79-8f3b-b26007b967de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-t2mvg\" (UID: \"18a6ea09-f03e-4b79-8f3b-b26007b967de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:09.879494 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.879444 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:09.970742 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:09.970694 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:10.156175 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.156011 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg"] Jul 22 12:49:10.159358 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:49:10.159318 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a6ea09_f03e_4b79_8f3b_b26007b967de.slice/crio-ea7a7e44508202b8c14d33b06bd859d34b8d85c0d43c83148a5d0505f21b5e2c WatchSource:0}: Error finding container ea7a7e44508202b8c14d33b06bd859d34b8d85c0d43c83148a5d0505f21b5e2c: Status 404 returned error can't find the container with id ea7a7e44508202b8c14d33b06bd859d34b8d85c0d43c83148a5d0505f21b5e2c Jul 22 12:49:10.162077 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.162045 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:49:10.162418 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.162114 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:49:10.162418 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.162140 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:49:10.550634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.550600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" event={"ID":"18a6ea09-f03e-4b79-8f3b-b26007b967de","Type":"ContainerStarted","Data":"9c810248356fdf3cdaab62b50d6ad0866f028149b72052f2f006227db93fec0a"} Jul 22 12:49:10.551035 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.550649 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" event={"ID":"18a6ea09-f03e-4b79-8f3b-b26007b967de","Type":"ContainerStarted","Data":"ea7a7e44508202b8c14d33b06bd859d34b8d85c0d43c83148a5d0505f21b5e2c"} Jul 22 12:49:10.575835 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.575776 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" podStartSLOduration=1.575752719 podStartE2EDuration="1.575752719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:10.573020338 +0000 UTC m=+489.660509520" watchObservedRunningTime="2026-07-22 12:49:10.575752719 +0000 UTC m=+489.663241904" Jul 22 12:49:10.971687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:10.971649 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:11.703187 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:11.703153 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:11.777691 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:11.777653 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:11.880410 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:11.880371 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:11.975971 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:11.975901 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:12.476584 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.476549 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:12.512960 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.512901 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:12.820283 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.820202 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:12.820283 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.820287 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:49:12.820685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.820306 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:12.820685 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.820468 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:12.823105 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.823084 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q5zrv\"" Jul 22 12:49:12.876372 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.876340 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjrr\" (UniqueName: \"kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr\") pod \"authorino-58cd955675-t52f8\" (UID: \"9eaf7f64-ef00-47f1-8e41-129e8f43a74f\") " pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:12.924285 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.924254 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:12.977276 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.977232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjrr\" (UniqueName: \"kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr\") pod \"authorino-58cd955675-t52f8\" (UID: \"9eaf7f64-ef00-47f1-8e41-129e8f43a74f\") " pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:12.986129 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:12.986091 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjrr\" (UniqueName: \"kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr\") pod \"authorino-58cd955675-t52f8\" (UID: \"9eaf7f64-ef00-47f1-8e41-129e8f43a74f\") " pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:13.129841 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:13.129755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:13.376014 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:13.375946 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:13.379102 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:49:13.379069 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eaf7f64_ef00_47f1_8e41_129e8f43a74f.slice/crio-0808a2f33fe2bd9980fba27e8f9e57cec8fe85e2b3457de0e7e1616fc7a21639 WatchSource:0}: Error finding container 0808a2f33fe2bd9980fba27e8f9e57cec8fe85e2b3457de0e7e1616fc7a21639: Status 404 returned error can't find the container with id 0808a2f33fe2bd9980fba27e8f9e57cec8fe85e2b3457de0e7e1616fc7a21639 Jul 22 12:49:13.565229 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:13.565195 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t52f8" event={"ID":"9eaf7f64-ef00-47f1-8e41-129e8f43a74f","Type":"ContainerStarted","Data":"0808a2f33fe2bd9980fba27e8f9e57cec8fe85e2b3457de0e7e1616fc7a21639"} Jul 22 12:49:16.584363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:16.584269 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t52f8" event={"ID":"9eaf7f64-ef00-47f1-8e41-129e8f43a74f","Type":"ContainerStarted","Data":"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b"} Jul 22 12:49:16.603104 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:16.603050 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-t52f8" podStartSLOduration=1.765480218 podStartE2EDuration="4.603033691s" totalImagesPullingTime="2.837553473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:16.602162148 +0000 UTC m=+495.689651329" watchObservedRunningTime="2026-07-22 12:49:16.603033691 +0000 UTC m=+495.690522871" Jul 22 12:49:17.243333 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:17.243291 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:18.592242 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:18.592187 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-t52f8" podUID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" containerName="authorino" containerID="cri-o://86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b" gracePeriod=30 Jul 22 12:49:18.867591 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:18.867521 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:18.931085 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:18.931051 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmjrr\" (UniqueName: \"kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr\") pod \"9eaf7f64-ef00-47f1-8e41-129e8f43a74f\" (UID: \"9eaf7f64-ef00-47f1-8e41-129e8f43a74f\") " Jul 22 12:49:18.933498 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:18.933473 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr" (OuterVolumeSpecName: "kube-api-access-dmjrr") pod "9eaf7f64-ef00-47f1-8e41-129e8f43a74f" (UID: "9eaf7f64-ef00-47f1-8e41-129e8f43a74f"). InnerVolumeSpecName "kube-api-access-dmjrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:49:19.031995 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.031951 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmjrr\" (UniqueName: \"kubernetes.io/projected/9eaf7f64-ef00-47f1-8e41-129e8f43a74f-kube-api-access-dmjrr\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:49:19.598359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.598319 2557 generic.go:356] "Generic (PLEG): container finished" podID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" containerID="86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b" exitCode=0 Jul 22 12:49:19.598359 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.598364 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-t52f8" Jul 22 12:49:19.598883 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.598386 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t52f8" event={"ID":"9eaf7f64-ef00-47f1-8e41-129e8f43a74f","Type":"ContainerDied","Data":"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b"} Jul 22 12:49:19.598883 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.598423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-t52f8" event={"ID":"9eaf7f64-ef00-47f1-8e41-129e8f43a74f","Type":"ContainerDied","Data":"0808a2f33fe2bd9980fba27e8f9e57cec8fe85e2b3457de0e7e1616fc7a21639"} Jul 22 12:49:19.598883 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.598445 2557 scope.go:117] "RemoveContainer" containerID="86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b" Jul 22 12:49:19.607140 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.607119 2557 scope.go:117] "RemoveContainer" containerID="86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b" Jul 22 12:49:19.607483 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:49:19.607460 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b\": container with ID starting with 86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b not found: ID does not exist" containerID="86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b" Jul 22 12:49:19.607543 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.607492 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b"} err="failed to get container status \"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b\": rpc error: code = NotFound desc = could not find container \"86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b\": container with ID starting with 86f723f099b3ed5b2b543f049b9e64ee74398eb43409aafae87acccc387fdd9b not found: ID does not exist" Jul 22 12:49:19.617373 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.617312 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:19.619429 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:19.619401 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-t52f8"] Jul 22 12:49:21.486017 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:21.485976 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" path="/var/lib/kubelet/pods/9eaf7f64-ef00-47f1-8e41-129e8f43a74f/volumes" Jul 22 12:49:29.819314 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:29.819280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:29.886269 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:29.886239 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:32.989632 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:32.989597 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:33.181247 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:33.181215 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:33.551387 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:33.551353 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:45.723415 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.723375 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:45.869248 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.869206 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:45.900897 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.900859 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:45.901228 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.901213 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" containerName="authorino" Jul 22 12:49:45.901271 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.901231 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" containerName="authorino" Jul 22 12:49:45.901306 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.901299 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eaf7f64-ef00-47f1-8e41-129e8f43a74f" containerName="authorino" Jul 22 12:49:45.907552 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.907532 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:45.910410 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.910375 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q5zrv\"" Jul 22 12:49:45.911956 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.911932 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:45.966471 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:45.966436 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxd8l\" (UniqueName: \"kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l\") pod \"authorino-5d4b67c446-s877p\" (UID: \"ada91783-db10-4a9e-8ee7-f31e83fbb6a6\") " pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:46.067147 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.067058 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxd8l\" (UniqueName: \"kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l\") pod \"authorino-5d4b67c446-s877p\" (UID: \"ada91783-db10-4a9e-8ee7-f31e83fbb6a6\") " pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:46.076177 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.076145 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxd8l\" (UniqueName: \"kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l\") pod \"authorino-5d4b67c446-s877p\" (UID: \"ada91783-db10-4a9e-8ee7-f31e83fbb6a6\") " pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:46.146288 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.146259 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:46.146531 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.146512 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:46.178392 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.178355 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf54f57bd-dw5xs"] Jul 22 12:49:46.203381 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.203342 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf54f57bd-dw5xs"] Jul 22 12:49:46.203639 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.203611 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.268844 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.268807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7b5\" (UniqueName: \"kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5\") pod \"authorino-7cf54f57bd-dw5xs\" (UID: \"52ecb030-7a34-4d2f-ad58-edf868e9542a\") " pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.285281 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.285253 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:46.287648 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:49:46.287621 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada91783_db10_4a9e_8ee7_f31e83fbb6a6.slice/crio-63987fc23b382165206dcf4878b152a26456ccf274e7f296aea28f2776191d49 WatchSource:0}: Error finding container 63987fc23b382165206dcf4878b152a26456ccf274e7f296aea28f2776191d49: Status 404 returned error can't find the container with id 63987fc23b382165206dcf4878b152a26456ccf274e7f296aea28f2776191d49 Jul 22 12:49:46.345102 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.345013 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cf54f57bd-dw5xs"] Jul 22 12:49:46.345285 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:49:46.345266 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dx7b5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" podUID="52ecb030-7a34-4d2f-ad58-edf868e9542a" Jul 22 12:49:46.369613 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.369576 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7b5\" (UniqueName: \"kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5\") pod \"authorino-7cf54f57bd-dw5xs\" (UID: \"52ecb030-7a34-4d2f-ad58-edf868e9542a\") " pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.372280 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.372255 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:49:46.379983 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.379959 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7b5\" (UniqueName: \"kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5\") pod \"authorino-7cf54f57bd-dw5xs\" (UID: \"52ecb030-7a34-4d2f-ad58-edf868e9542a\") " pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.395126 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.395100 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:49:46.395276 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.395205 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.397667 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.397645 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 12:49:46.470687 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.470648 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.470890 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.470701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkd4\" (UniqueName: \"kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.572063 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.572023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.572238 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.572096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkd4\" (UniqueName: \"kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.574466 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.574436 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.580925 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.580904 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkd4\" (UniqueName: \"kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4\") pod \"authorino-7c47b75964-7h6xk\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.705071 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.705045 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:49:46.727867 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.727832 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s877p" event={"ID":"ada91783-db10-4a9e-8ee7-f31e83fbb6a6","Type":"ContainerStarted","Data":"63987fc23b382165206dcf4878b152a26456ccf274e7f296aea28f2776191d49"} Jul 22 12:49:46.728207 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.727877 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.746087 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.746061 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:46.849119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.849091 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:49:46.853275 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:49:46.853252 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab790130_17f2_497c_b20a_44d56ef8cba3.slice/crio-2c7d5e0b79dc1473cdd4371c0e0a7ca5ce498c4807c2bfa518982c8ffe5ecae8 WatchSource:0}: Error finding container 2c7d5e0b79dc1473cdd4371c0e0a7ca5ce498c4807c2bfa518982c8ffe5ecae8: Status 404 returned error can't find the container with id 2c7d5e0b79dc1473cdd4371c0e0a7ca5ce498c4807c2bfa518982c8ffe5ecae8 Jul 22 12:49:46.874886 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.874858 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx7b5\" (UniqueName: \"kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5\") pod \"52ecb030-7a34-4d2f-ad58-edf868e9542a\" (UID: \"52ecb030-7a34-4d2f-ad58-edf868e9542a\") " Jul 22 12:49:46.876855 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.876828 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5" (OuterVolumeSpecName: "kube-api-access-dx7b5") pod "52ecb030-7a34-4d2f-ad58-edf868e9542a" (UID: "52ecb030-7a34-4d2f-ad58-edf868e9542a"). InnerVolumeSpecName "kube-api-access-dx7b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:49:46.976469 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:46.976433 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7b5\" (UniqueName: \"kubernetes.io/projected/52ecb030-7a34-4d2f-ad58-edf868e9542a-kube-api-access-dx7b5\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:49:47.734057 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.734017 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c47b75964-7h6xk" event={"ID":"ab790130-17f2-497c-b20a-44d56ef8cba3","Type":"ContainerStarted","Data":"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d"} Jul 22 12:49:47.734569 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.734070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c47b75964-7h6xk" event={"ID":"ab790130-17f2-497c-b20a-44d56ef8cba3","Type":"ContainerStarted","Data":"2c7d5e0b79dc1473cdd4371c0e0a7ca5ce498c4807c2bfa518982c8ffe5ecae8"} Jul 22 12:49:47.735483 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.735457 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s877p" event={"ID":"ada91783-db10-4a9e-8ee7-f31e83fbb6a6","Type":"ContainerStarted","Data":"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9"} Jul 22 12:49:47.735574 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.735497 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf54f57bd-dw5xs" Jul 22 12:49:47.735574 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.735495 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-s877p" podUID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" containerName="authorino" containerID="cri-o://da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9" gracePeriod=30 Jul 22 12:49:47.752554 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.752505 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c47b75964-7h6xk" podStartSLOduration=1.274972544 podStartE2EDuration="1.752488322s" totalImagesPullingTime="477.515778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:47.752001685 +0000 UTC m=+526.839490861" watchObservedRunningTime="2026-07-22 12:49:47.752488322 +0000 UTC m=+526.839977481" Jul 22 12:49:47.777311 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.777279 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cf54f57bd-dw5xs"] Jul 22 12:49:47.784747 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.784719 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cf54f57bd-dw5xs"] Jul 22 12:49:47.805328 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.805267 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-s877p" podStartSLOduration=2.210404631 podStartE2EDuration="2.805243179s" totalImagesPullingTime="594.838548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:47.804183205 +0000 UTC m=+526.891672378" watchObservedRunningTime="2026-07-22 12:49:47.805243179 +0000 UTC m=+526.892732362" Jul 22 12:49:47.965078 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:47.965057 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:48.085502 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.085477 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxd8l\" (UniqueName: \"kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l\") pod \"ada91783-db10-4a9e-8ee7-f31e83fbb6a6\" (UID: \"ada91783-db10-4a9e-8ee7-f31e83fbb6a6\") " Jul 22 12:49:48.087482 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.087458 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l" (OuterVolumeSpecName: "kube-api-access-nxd8l") pod "ada91783-db10-4a9e-8ee7-f31e83fbb6a6" (UID: "ada91783-db10-4a9e-8ee7-f31e83fbb6a6"). InnerVolumeSpecName "kube-api-access-nxd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:49:48.186237 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.186192 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxd8l\" (UniqueName: \"kubernetes.io/projected/ada91783-db10-4a9e-8ee7-f31e83fbb6a6-kube-api-access-nxd8l\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:49:48.741192 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.741153 2557 generic.go:356] "Generic (PLEG): container finished" podID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" containerID="da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9" exitCode=0 Jul 22 12:49:48.741634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.741197 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s877p" Jul 22 12:49:48.741634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.741236 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s877p" event={"ID":"ada91783-db10-4a9e-8ee7-f31e83fbb6a6","Type":"ContainerDied","Data":"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9"} Jul 22 12:49:48.741634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.741279 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s877p" event={"ID":"ada91783-db10-4a9e-8ee7-f31e83fbb6a6","Type":"ContainerDied","Data":"63987fc23b382165206dcf4878b152a26456ccf274e7f296aea28f2776191d49"} Jul 22 12:49:48.741634 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.741303 2557 scope.go:117] "RemoveContainer" containerID="da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9" Jul 22 12:49:48.749517 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.749499 2557 scope.go:117] "RemoveContainer" containerID="da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9" Jul 22 12:49:48.749814 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:49:48.749794 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9\": container with ID starting with da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9 not found: ID does not exist" containerID="da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9" Jul 22 12:49:48.749878 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.749823 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9"} err="failed to get container status \"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9\": rpc error: code = NotFound desc = could not find container \"da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9\": container with ID starting with da6370ce50cf74962de796dbca6d8c187292fc66a856606b688f61ef603eb2c9 not found: ID does not exist" Jul 22 12:49:48.763958 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.763923 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:48.769371 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:48.769345 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s877p"] Jul 22 12:49:49.485397 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:49.485366 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ecb030-7a34-4d2f-ad58-edf868e9542a" path="/var/lib/kubelet/pods/52ecb030-7a34-4d2f-ad58-edf868e9542a/volumes" Jul 22 12:49:49.485595 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:49.485584 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" path="/var/lib/kubelet/pods/ada91783-db10-4a9e-8ee7-f31e83fbb6a6/volumes" Jul 22 12:49:52.686159 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:52.686073 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:52.765137 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:52.765108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:52.839640 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:52.839605 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:56.061363 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:56.061319 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:49:56.157013 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:49:56.156977 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:27.754488 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:27.754453 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:27.863658 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:27.863629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:28.272521 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:28.272488 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:28.493840 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:28.493808 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:28.880933 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:28.880900 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:28.986099 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:28.986063 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:36.950286 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.950254 2557 generic.go:356] "Generic (PLEG): container finished" podID="18a6ea09-f03e-4b79-8f3b-b26007b967de" containerID="9c810248356fdf3cdaab62b50d6ad0866f028149b72052f2f006227db93fec0a" exitCode=0 Jul 22 12:50:36.950843 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.950327 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" event={"ID":"18a6ea09-f03e-4b79-8f3b-b26007b967de","Type":"ContainerDied","Data":"9c810248356fdf3cdaab62b50d6ad0866f028149b72052f2f006227db93fec0a"} Jul 22 12:50:36.950843 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.950682 2557 scope.go:117] "RemoveContainer" containerID="9c810248356fdf3cdaab62b50d6ad0866f028149b72052f2f006227db93fec0a" Jul 22 12:50:36.951643 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.951620 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:50:36.951769 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.951683 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:50:36.951769 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:36.951734 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 12:50:37.957036 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:37.957003 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" event={"ID":"18a6ea09-f03e-4b79-8f3b-b26007b967de","Type":"ContainerStarted","Data":"0c2eec0b7fac04b3aed70e4876adde98410af6327715f8e50ce51a6fbbf916c2"} Jul 22 12:50:37.971441 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:37.971411 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:50:38.972135 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:38.972101 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-t2mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 12:50:38.972521 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:38.972170 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" podUID="18a6ea09-f03e-4b79-8f3b-b26007b967de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 12:50:39.193081 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:39.193044 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:39.295949 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:39.295856 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:39.971151 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:39.971113 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-t2mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 12:50:39.971325 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:39.971170 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" podUID="18a6ea09-f03e-4b79-8f3b-b26007b967de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 12:50:40.971522 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:40.971484 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-t2mvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 22 12:50:40.971952 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:40.971538 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" podUID="18a6ea09-f03e-4b79-8f3b-b26007b967de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 22 12:50:41.975255 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:41.975226 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:50:41.975630 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:41.975499 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:50:41.976234 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:41.976217 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-t2mvg" Jul 22 12:50:48.373859 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.373813 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm"] Jul 22 12:50:48.374252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.374138 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" containerName="authorino" Jul 22 12:50:48.374252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.374148 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" containerName="authorino" Jul 22 12:50:48.374252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.374207 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada91783-db10-4a9e-8ee7-f31e83fbb6a6" containerName="authorino" Jul 22 12:50:48.375930 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.375912 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.379889 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.379863 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nqcsn\"" Jul 22 12:50:48.379889 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.379886 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 12:50:48.380119 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.379897 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 12:50:48.380233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.380217 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 12:50:48.393035 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.393001 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm"] Jul 22 12:50:48.416233 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.416194 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:48.527117 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.527117 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527117 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdkj\" (UniqueName: \"kubernetes.io/projected/a643b935-b0ae-476c-a0da-424e82a615f7-kube-api-access-wqdkj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.527334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.527334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527195 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.527334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a643b935-b0ae-476c-a0da-424e82a615f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.527334 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.527264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.628756 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628628 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a643b935-b0ae-476c-a0da-424e82a615f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.628756 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.628756 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdkj\" (UniqueName: \"kubernetes.io/projected/a643b935-b0ae-476c-a0da-424e82a615f7-kube-api-access-wqdkj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629044 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.628850 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.629071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.629105 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.629191 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.629145 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.631014 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.630994 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a643b935-b0ae-476c-a0da-424e82a615f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.631305 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.631289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a643b935-b0ae-476c-a0da-424e82a615f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.640342 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.640307 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdkj\" (UniqueName: \"kubernetes.io/projected/a643b935-b0ae-476c-a0da-424e82a615f7-kube-api-access-wqdkj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm\" (UID: \"a643b935-b0ae-476c-a0da-424e82a615f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.686290 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.686230 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:48.836366 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:48.836341 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm"] Jul 22 12:50:48.838913 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:50:48.838886 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda643b935_b0ae_476c_a0da_424e82a615f7.slice/crio-580b8c8f1d6e14580e40724356f10e7d7de7806c58195c7eae84d9f9acc04d9f WatchSource:0}: Error finding container 580b8c8f1d6e14580e40724356f10e7d7de7806c58195c7eae84d9f9acc04d9f: Status 404 returned error can't find the container with id 580b8c8f1d6e14580e40724356f10e7d7de7806c58195c7eae84d9f9acc04d9f Jul 22 12:50:49.006892 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:49.006848 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" event={"ID":"a643b935-b0ae-476c-a0da-424e82a615f7","Type":"ContainerStarted","Data":"580b8c8f1d6e14580e40724356f10e7d7de7806c58195c7eae84d9f9acc04d9f"} Jul 22 12:50:50.141976 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:50.141927 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:50.241100 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:50.241065 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:51.016751 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:51.016703 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" event={"ID":"a643b935-b0ae-476c-a0da-424e82a615f7","Type":"ContainerStarted","Data":"052985539fa016fe27be8202c41c78e91b63871b781de113ab64e7bedae209a4"} Jul 22 12:50:51.037643 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:51.037596 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" podStartSLOduration=1.879605661 podStartE2EDuration="3.037578211s" totalImagesPullingTime="1.15797255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:50:51.036961385 +0000 UTC m=+590.124450565" watchObservedRunningTime="2026-07-22 12:50:51.037578211 +0000 UTC m=+590.125067391" Jul 22 12:50:52.315205 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:52.315169 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:50:58.687396 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:58.687345 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:58.687964 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:58.687430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:58.699405 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:58.699381 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:50:59.061227 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:50:59.061200 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm" Jul 22 12:51:08.529000 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:08.528964 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:10.490574 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:10.490541 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:13.960475 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:13.960439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:35.920659 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:35.920565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:43.870288 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:43.870247 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t"] Jul 22 12:51:43.873908 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:43.873884 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:43.877197 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:43.877171 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 12:51:43.885908 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:43.885885 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t"] Jul 22 12:51:43.914236 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:43.914197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:44.013441 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013402 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.013624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fr6w\" (UniqueName: \"kubernetes.io/projected/d44a70c1-d522-4be1-9238-c3087ab8ecaf-kube-api-access-6fr6w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.013624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.013624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013570 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.013624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.013624 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.013610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.114795 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.114758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fr6w\" (UniqueName: \"kubernetes.io/projected/d44a70c1-d522-4be1-9238-c3087ab8ecaf-kube-api-access-6fr6w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.114795 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.114799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.114825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.114851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115046 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.114989 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115196 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.115087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.115272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115345 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.115301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.115453 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.115430 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.117093 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.117074 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d44a70c1-d522-4be1-9238-c3087ab8ecaf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.117482 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.117461 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d44a70c1-d522-4be1-9238-c3087ab8ecaf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.135802 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.135736 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fr6w\" (UniqueName: \"kubernetes.io/projected/d44a70c1-d522-4be1-9238-c3087ab8ecaf-kube-api-access-6fr6w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-s525t\" (UID: \"d44a70c1-d522-4be1-9238-c3087ab8ecaf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.186122 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.186090 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:44.355722 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.355671 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t"] Jul 22 12:51:44.358487 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:51:44.358461 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44a70c1_d522_4be1_9238_c3087ab8ecaf.slice/crio-569ab25cc80aab0234a89666c96160a939e6aac1fcd0c1caf5cb7f455409b0a0 WatchSource:0}: Error finding container 569ab25cc80aab0234a89666c96160a939e6aac1fcd0c1caf5cb7f455409b0a0: Status 404 returned error can't find the container with id 569ab25cc80aab0234a89666c96160a939e6aac1fcd0c1caf5cb7f455409b0a0 Jul 22 12:51:44.359779 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:44.359763 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:51:45.252898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:45.252856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" event={"ID":"d44a70c1-d522-4be1-9238-c3087ab8ecaf","Type":"ContainerStarted","Data":"a977b938421b565a6df8f80abc3fb371b62f30a4247909391c2dd3c042e52c2a"} Jul 22 12:51:45.252898 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:45.252905 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" event={"ID":"d44a70c1-d522-4be1-9238-c3087ab8ecaf","Type":"ContainerStarted","Data":"569ab25cc80aab0234a89666c96160a939e6aac1fcd0c1caf5cb7f455409b0a0"} Jul 22 12:51:45.292326 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:45.292275 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" podStartSLOduration=2.075364062 podStartE2EDuration="2.292257498s" totalImagesPullingTime="216.893436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:51:45.287193468 +0000 UTC m=+644.374682662" watchObservedRunningTime="2026-07-22 12:51:45.292257498 +0000 UTC m=+644.379746678" Jul 22 12:51:47.371223 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:47.371174 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:51:54.186579 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:54.186533 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:54.187033 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:54.186634 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:54.199135 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:54.199108 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:51:54.304056 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:51:54.304027 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-s525t" Jul 22 12:52:03.443209 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:03.443173 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:52:46.006590 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.006556 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c859549df-g727n"] Jul 22 12:52:46.010017 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.010000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.021118 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.021094 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c859549df-g727n"] Jul 22 12:52:46.048021 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.047982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8bq\" (UniqueName: \"kubernetes.io/projected/2be1a884-af4b-4ded-9107-15a956f2c509-kube-api-access-6p8bq\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.048200 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.048118 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be1a884-af4b-4ded-9107-15a956f2c509-tls-cert\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.149252 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.149213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8bq\" (UniqueName: \"kubernetes.io/projected/2be1a884-af4b-4ded-9107-15a956f2c509-kube-api-access-6p8bq\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.149444 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.149324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be1a884-af4b-4ded-9107-15a956f2c509-tls-cert\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.151780 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.151756 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2be1a884-af4b-4ded-9107-15a956f2c509-tls-cert\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.157730 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.157694 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8bq\" (UniqueName: \"kubernetes.io/projected/2be1a884-af4b-4ded-9107-15a956f2c509-kube-api-access-6p8bq\") pod \"authorino-c859549df-g727n\" (UID: \"2be1a884-af4b-4ded-9107-15a956f2c509\") " pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.318851 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.318755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c859549df-g727n" Jul 22 12:52:46.414084 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.414013 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:52:46.459543 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.459514 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c859549df-g727n"] Jul 22 12:52:46.462417 ip-10-0-137-115 kubenswrapper[2557]: W0722 12:52:46.462386 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be1a884_af4b_4ded_9107_15a956f2c509.slice/crio-6e543a200b065188bb6f8485efa6ba88a9cae8ef6916fa32c278bbd5041d4c61 WatchSource:0}: Error finding container 6e543a200b065188bb6f8485efa6ba88a9cae8ef6916fa32c278bbd5041d4c61: Status 404 returned error can't find the container with id 6e543a200b065188bb6f8485efa6ba88a9cae8ef6916fa32c278bbd5041d4c61 Jul 22 12:52:46.510506 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.510473 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:52:46.520889 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:46.520861 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c859549df-g727n" event={"ID":"2be1a884-af4b-4ded-9107-15a956f2c509","Type":"ContainerStarted","Data":"6e543a200b065188bb6f8485efa6ba88a9cae8ef6916fa32c278bbd5041d4c61"} Jul 22 12:52:47.527480 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.527443 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c859549df-g727n" event={"ID":"2be1a884-af4b-4ded-9107-15a956f2c509","Type":"ContainerStarted","Data":"9d16e96bf0c316b2e18de429852c04b41f001f7b9f4cb9d9edd103ecec38396f"} Jul 22 12:52:47.580986 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.580920 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c859549df-g727n" podStartSLOduration=2.172177767 podStartE2EDuration="2.580894491s" totalImagesPullingTime="408.716724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:52:47.576743968 +0000 UTC m=+706.664233150" watchObservedRunningTime="2026-07-22 12:52:47.580894491 +0000 UTC m=+706.668383674" Jul 22 12:52:47.695737 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.695673 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:52:47.695948 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.695919 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c47b75964-7h6xk" podUID="ab790130-17f2-497c-b20a-44d56ef8cba3" containerName="authorino" containerID="cri-o://d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d" gracePeriod=30 Jul 22 12:52:47.766840 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.766781 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:52:47.907075 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.907040 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:52:47.967398 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:47.967368 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:52:48.068216 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.068139 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert\") pod \"ab790130-17f2-497c-b20a-44d56ef8cba3\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " Jul 22 12:52:48.068361 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.068242 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmkd4\" (UniqueName: \"kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4\") pod \"ab790130-17f2-497c-b20a-44d56ef8cba3\" (UID: \"ab790130-17f2-497c-b20a-44d56ef8cba3\") " Jul 22 12:52:48.070297 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.070249 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4" (OuterVolumeSpecName: "kube-api-access-pmkd4") pod "ab790130-17f2-497c-b20a-44d56ef8cba3" (UID: "ab790130-17f2-497c-b20a-44d56ef8cba3"). InnerVolumeSpecName "kube-api-access-pmkd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:52:48.077368 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.077339 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ab790130-17f2-497c-b20a-44d56ef8cba3" (UID: "ab790130-17f2-497c-b20a-44d56ef8cba3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:52:48.169290 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.169250 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkd4\" (UniqueName: \"kubernetes.io/projected/ab790130-17f2-497c-b20a-44d56ef8cba3-kube-api-access-pmkd4\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:52:48.169290 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.169291 2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab790130-17f2-497c-b20a-44d56ef8cba3-tls-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 12:52:48.533895 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.533861 2557 generic.go:356] "Generic (PLEG): container finished" podID="ab790130-17f2-497c-b20a-44d56ef8cba3" containerID="d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d" exitCode=0 Jul 22 12:52:48.534320 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.533908 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c47b75964-7h6xk" Jul 22 12:52:48.534320 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.533951 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c47b75964-7h6xk" event={"ID":"ab790130-17f2-497c-b20a-44d56ef8cba3","Type":"ContainerDied","Data":"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d"} Jul 22 12:52:48.534320 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.533997 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c47b75964-7h6xk" event={"ID":"ab790130-17f2-497c-b20a-44d56ef8cba3","Type":"ContainerDied","Data":"2c7d5e0b79dc1473cdd4371c0e0a7ca5ce498c4807c2bfa518982c8ffe5ecae8"} Jul 22 12:52:48.534320 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.534018 2557 scope.go:117] "RemoveContainer" containerID="d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d" Jul 22 12:52:48.543198 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.543182 2557 scope.go:117] "RemoveContainer" containerID="d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d" Jul 22 12:52:48.543473 ip-10-0-137-115 kubenswrapper[2557]: E0722 12:52:48.543453 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d\": container with ID starting with d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d not found: ID does not exist" containerID="d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d" Jul 22 12:52:48.543541 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.543480 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d"} err="failed to get container status \"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d\": rpc error: code = NotFound desc = could not find container \"d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d\": container with ID starting with d29a2559471449943ec8de05d51917aa29fd38494d9d4b407f23e8e88333367d not found: ID does not exist" Jul 22 12:52:48.568318 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.568290 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:52:48.581734 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:48.581670 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c47b75964-7h6xk"] Jul 22 12:52:49.485081 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:52:49.485050 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab790130-17f2-497c-b20a-44d56ef8cba3" path="/var/lib/kubelet/pods/ab790130-17f2-497c-b20a-44d56ef8cba3/volumes" Jul 22 12:53:39.322603 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:53:39.322560 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:54:05.654508 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:54:05.654474 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:54:53.268521 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:54:53.268484 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:54:54.511264 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:54:54.511222 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:15.769388 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:15.769351 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:15.870555 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:15.870517 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:28.337908 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:28.337874 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:28.436018 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:28.435983 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:28.590455 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:28.590361 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:34.678876 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:34.678843 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:34.820617 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:34.820572 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:35.010688 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:35.010629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:35.110694 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:35.110664 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:41.121315 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:41.121277 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:55:41.212247 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:55:41.212218 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:57:48.652996 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:57:48.652957 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:57:51.534874 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:57:51.534835 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:58:05.201512 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:58:05.201473 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 12:58:05.302605 ip-10-0-137-115 kubenswrapper[2557]: I0722 12:58:05.302570 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:00:08.810012 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:00:08.809969 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:01:14.688963 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:01:14.688926 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:03:25.772989 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.772891 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:03:25.773580 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.773413 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab790130-17f2-497c-b20a-44d56ef8cba3" containerName="authorino" Jul 22 13:03:25.773580 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.773431 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab790130-17f2-497c-b20a-44d56ef8cba3" containerName="authorino" Jul 22 13:03:25.773580 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.773511 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab790130-17f2-497c-b20a-44d56ef8cba3" containerName="authorino" Jul 22 13:03:25.776556 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.776533 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.779236 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.779211 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-20f81251-kserve-self-signed-certs\"" Jul 22 13:03:25.792357 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.792320 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:03:25.841452 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.841422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:03:25.872952 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.872908 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.873133 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.872966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.873133 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.873034 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.873133 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.873092 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2vg\" (UniqueName: \"kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.873263 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.873138 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.873263 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.873160 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974041 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.973997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974041 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2vg\" (UniqueName: \"kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974313 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974071 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974313 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974313 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974313 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974526 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974577 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974545 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.974577 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.974556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.976406 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.976383 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.976617 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.976599 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:25.983366 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:25.983345 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2vg\" (UniqueName: \"kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg\") pod \"e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:26.086963 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:26.086863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:26.250602 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:26.250571 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:03:26.252225 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:03:26.252199 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b25f9f2_bfee_43af_b6b3_2ead051c6929.slice/crio-d7dd2d6c55fb921008d10c3ca35cb37ae75865f9ba1edeb33c13fc0557669092 WatchSource:0}: Error finding container d7dd2d6c55fb921008d10c3ca35cb37ae75865f9ba1edeb33c13fc0557669092: Status 404 returned error can't find the container with id d7dd2d6c55fb921008d10c3ca35cb37ae75865f9ba1edeb33c13fc0557669092 Jul 22 13:03:26.431682 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:26.431590 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" event={"ID":"0b25f9f2-bfee-43af-b6b3-2ead051c6929","Type":"ContainerStarted","Data":"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def"} Jul 22 13:03:26.431682 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:26.431634 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" event={"ID":"0b25f9f2-bfee-43af-b6b3-2ead051c6929","Type":"ContainerStarted","Data":"d7dd2d6c55fb921008d10c3ca35cb37ae75865f9ba1edeb33c13fc0557669092"} Jul 22 13:03:26.482229 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:26.482173 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" podStartSLOduration=1.482155582 podStartE2EDuration="1.482155582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:03:26.481644051 +0000 UTC m=+1345.569133231" watchObservedRunningTime="2026-07-22 13:03:26.482155582 +0000 UTC m=+1345.569644762" Jul 22 13:03:36.087958 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:36.087851 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:36.088354 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:36.087958 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:36.100325 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:36.100299 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:36.487177 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:36.487151 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:03:37.892146 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:03:37.892115 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:04:03.356466 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:03.356425 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:04:03.387841 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:03.387806 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:04:04.324555 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:04.324520 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:04:14.898245 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:14.898210 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:04:18.376914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.376878 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" podUID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" containerName="main" containerID="cri-o://a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def" gracePeriod=45 Jul 22 13:04:18.642380 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.642305 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:04:18.669896 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.669860 2557 generic.go:356] "Generic (PLEG): container finished" podID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" containerID="a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def" exitCode=0 Jul 22 13:04:18.670112 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.669943 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" Jul 22 13:04:18.670112 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.669906 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" event={"ID":"0b25f9f2-bfee-43af-b6b3-2ead051c6929","Type":"ContainerDied","Data":"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def"} Jul 22 13:04:18.670112 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.670026 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq" event={"ID":"0b25f9f2-bfee-43af-b6b3-2ead051c6929","Type":"ContainerDied","Data":"d7dd2d6c55fb921008d10c3ca35cb37ae75865f9ba1edeb33c13fc0557669092"} Jul 22 13:04:18.670112 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.670099 2557 scope.go:117] "RemoveContainer" containerID="a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def" Jul 22 13:04:18.681464 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.681438 2557 scope.go:117] "RemoveContainer" containerID="a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def" Jul 22 13:04:18.681832 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:04:18.681808 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def\": container with ID starting with a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def not found: ID does not exist" containerID="a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def" Jul 22 13:04:18.681893 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.681842 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def"} err="failed to get container status \"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def\": rpc error: code = NotFound desc = could not find container \"a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def\": container with ID starting with a495e7b289ea989e1bb972f53fb41b5de2a2f6a2ccceeb9d2a69cd641fd83def not found: ID does not exist" Jul 22 13:04:18.755951 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.755911 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm2vg\" (UniqueName: \"kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.755968 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756015 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756112 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756166 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756236 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home" (OuterVolumeSpecName: "home") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:18.756277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756250 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:18.756277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756257 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs\") pod \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\" (UID: \"0b25f9f2-bfee-43af-b6b3-2ead051c6929\") " Jul 22 13:04:18.756529 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756427 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:18.756596 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756581 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.756653 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756598 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.756653 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.756609 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.758409 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.758382 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm" (OuterVolumeSpecName: "dshm") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:18.758535 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.758438 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:04:18.758535 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.758475 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg" (OuterVolumeSpecName: "kube-api-access-sm2vg") pod "0b25f9f2-bfee-43af-b6b3-2ead051c6929" (UID: "0b25f9f2-bfee-43af-b6b3-2ead051c6929"). InnerVolumeSpecName "kube-api-access-sm2vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:04:18.857646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.857606 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b25f9f2-bfee-43af-b6b3-2ead051c6929-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.857646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.857641 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2vg\" (UniqueName: \"kubernetes.io/projected/0b25f9f2-bfee-43af-b6b3-2ead051c6929-kube-api-access-sm2vg\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.857646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.857653 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b25f9f2-bfee-43af-b6b3-2ead051c6929-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:04:18.992328 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.992288 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:04:18.996531 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:18.996498 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-20f81251-kserve-84f576db8c-89qmq"] Jul 22 13:04:19.486007 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:19.485971 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" path="/var/lib/kubelet/pods/0b25f9f2-bfee-43af-b6b3-2ead051c6929/volumes" Jul 22 13:04:54.389253 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:54.389170 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:04:54.467742 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:04:54.467696 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:06:28.811660 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:06:28.811579 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:06:28.958469 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:06:28.958436 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:06:30.802277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:06:30.802244 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:19.270273 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:19.270236 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:39.005096 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:39.005059 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:42.373959 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:42.373922 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:42.546170 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:42.546132 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:42.915811 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:42.915773 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:49.162625 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.162589 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:49.265432 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.265396 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:49.509767 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.509735 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:49.759106 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.759070 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:07:49.759561 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.759540 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" containerName="main" Jul 22 13:07:49.759635 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.759565 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" containerName="main" Jul 22 13:07:49.759673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.759643 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b25f9f2-bfee-43af-b6b3-2ead051c6929" containerName="main" Jul 22 13:07:49.762954 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.762908 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.766251 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.766234 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e1001375-openshift-default-dockercfg-mpkfz\"" Jul 22 13:07:49.788652 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.788618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:07:49.797638 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.797609 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:49.881895 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.881860 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwtr\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.881895 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.881907 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.881938 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.881968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.881996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.882085 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.882139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882372 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.882184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.882372 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.882280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983555 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983496 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983637 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983683 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwtr\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.983778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983770 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984057 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983802 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984057 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984057 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.983884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984254 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.984073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984551 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.984522 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984551 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.984543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984685 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.984591 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.984685 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.984600 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.986138 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.986116 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.986213 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.986157 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.992677 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.992656 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:49.992787 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:49.992676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwtr\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr\") pod \"e2e-ait-e1001375-openshift-default-55996fdcf-7rm22\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:50.073069 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.072956 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:50.225201 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.225166 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:07:50.227177 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:07:50.227146 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeba09ea_0135_4f99_8abc_1e58a150e068.slice/crio-75208c47cd62c418d6995c16fb82ab6a59d453f12fc1fe7b3f27a6616332b2ce WatchSource:0}: Error finding container 75208c47cd62c418d6995c16fb82ab6a59d453f12fc1fe7b3f27a6616332b2ce: Status 404 returned error can't find the container with id 75208c47cd62c418d6995c16fb82ab6a59d453f12fc1fe7b3f27a6616332b2ce Jul 22 13:07:50.229901 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.229862 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:07:50.230001 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.229952 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:07:50.230001 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.229993 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:07:50.628435 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.628380 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerStarted","Data":"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb"} Jul 22 13:07:50.628617 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.628459 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerStarted","Data":"75208c47cd62c418d6995c16fb82ab6a59d453f12fc1fe7b3f27a6616332b2ce"} Jul 22 13:07:50.652494 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.652453 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:50.656311 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:50.656259 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podStartSLOduration=1.656238995 podStartE2EDuration="1.656238995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:07:50.654366977 +0000 UTC m=+1609.741856162" watchObservedRunningTime="2026-07-22 13:07:50.656238995 +0000 UTC m=+1609.743728176" Jul 22 13:07:51.073888 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:51.073845 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:51.079094 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:51.079058 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:51.633390 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:51.633362 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:51.634583 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:51.634554 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:07:52.016633 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:52.016582 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:52.111519 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:52.111438 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:53.215728 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:53.215677 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:07:53.355238 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:07:53.355201 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:03.691805 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.691765 2557 generic.go:356] "Generic (PLEG): container finished" podID="beba09ea-0135-4f99-8abc-1e58a150e068" containerID="30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb" exitCode=0 Jul 22 13:08:03.692353 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.691843 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerDied","Data":"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb"} Jul 22 13:08:03.692434 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.692414 2557 scope.go:117] "RemoveContainer" containerID="30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb" Jul 22 13:08:03.693516 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.693486 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:03.693632 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.693557 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:03.693632 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:03.693586 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:04.699538 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:04.699485 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerStarted","Data":"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75"} Jul 22 13:08:05.073845 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:05.073751 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:05.075064 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:05.075032 2557 patch_prober.go:28] interesting pod/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 13:08:05.075225 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:05.075093 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 13:08:06.074019 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:06.073974 2557 patch_prober.go:28] interesting pod/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 13:08:06.074425 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:06.074040 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 13:08:06.633376 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:06.633343 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:07.074274 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:07.074234 2557 patch_prober.go:28] interesting pod/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 13:08:07.074681 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:07.074300 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 13:08:08.074051 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:08.074011 2557 patch_prober.go:28] interesting pod/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 13:08:08.074247 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:08.074071 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 13:08:09.078211 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:09.078175 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:09.079170 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:09.079150 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:38.874750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:38.874677 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:43.347830 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:43.347791 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:43.511377 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:43.511346 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:08:43.511586 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:43.511566 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" containerID="cri-o://e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75" gracePeriod=30 Jul 22 13:08:43.539299 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:43.539265 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:43.811065 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:43.811026 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:44.093535 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.093433 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:44.206083 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.206053 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:44.327800 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.327765 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:44.656694 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.656659 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:08:44.658856 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.658836 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.661967 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.661948 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-879879e7-gw-openshift-default-dockercfg-xbntn\"" Jul 22 13:08:44.678565 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.678526 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:08:44.690774 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.690737 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:44.700936 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.700901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701101 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.700941 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701101 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.700973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701101 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701227 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701227 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701205 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701306 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdfh\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701343 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701325 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.701384 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.701355 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802340 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802305 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802340 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802395 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdfh\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802741 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802824 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.802921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.802872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.803193 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.803176 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.803248 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.803221 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.803511 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.803488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.804852 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.804825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.805091 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.805075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.814977 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.814949 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.815566 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.815543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdfh\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh\") pod \"e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:44.969682 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:44.969635 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:45.033242 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.033156 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:45.113884 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.113856 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:08:45.115776 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:08:45.115752 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe235990_453b_4d99_b891_baae3ee68a88.slice/crio-98c4291e50189cff69752e846dd3282978ce12e1a4392da1d9a1209281f1ede2 WatchSource:0}: Error finding container 98c4291e50189cff69752e846dd3282978ce12e1a4392da1d9a1209281f1ede2: Status 404 returned error can't find the container with id 98c4291e50189cff69752e846dd3282978ce12e1a4392da1d9a1209281f1ede2 Jul 22 13:08:45.117911 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.117881 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:45.118008 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.117960 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:45.118008 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.118003 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:08:45.905082 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.905046 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerStarted","Data":"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0"} Jul 22 13:08:45.905497 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.905098 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerStarted","Data":"98c4291e50189cff69752e846dd3282978ce12e1a4392da1d9a1209281f1ede2"} Jul 22 13:08:45.930871 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.930823 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podStartSLOduration=1.930807289 podStartE2EDuration="1.930807289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:08:45.928586254 +0000 UTC m=+1665.016075434" watchObservedRunningTime="2026-07-22 13:08:45.930807289 +0000 UTC m=+1665.018296470" Jul 22 13:08:45.970494 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:45.970456 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:46.975084 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:46.975054 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:47.520965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:47.520902 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:47.621160 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:47.621110 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:47.720928 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:47.720893 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:47.915080 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:47.914985 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:47.916047 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:47.916030 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:08:48.074933 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.074900 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:48.176024 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.175939 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:08:48.756725 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.756686 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:48.843139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843033 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843094 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843121 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843410 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843167 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843410 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843196 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843410 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843232 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kwtr\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843410 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843292 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843410 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843352 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843403 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:08:48.843782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843434 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket\") pod \"beba09ea-0135-4f99-8abc-1e58a150e068\" (UID: \"beba09ea-0135-4f99-8abc-1e58a150e068\") " Jul 22 13:08:48.843782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843625 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:08:48.843782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843749 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:08:48.843782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843776 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.844052 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843826 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-workload-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.844052 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843872 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:08:48.844052 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.843903 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data" (OuterVolumeSpecName: "istio-data") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:08:48.845835 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.845801 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token" (OuterVolumeSpecName: "istio-token") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:08:48.845986 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.845959 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:08:48.846035 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.845998 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:08:48.846035 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.846017 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr" (OuterVolumeSpecName: "kube-api-access-7kwtr") pod "beba09ea-0135-4f99-8abc-1e58a150e068" (UID: "beba09ea-0135-4f99-8abc-1e58a150e068"). InnerVolumeSpecName "kube-api-access-7kwtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:08:48.921044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.921006 2557 generic.go:356] "Generic (PLEG): container finished" podID="beba09ea-0135-4f99-8abc-1e58a150e068" containerID="e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75" exitCode=0 Jul 22 13:08:48.921222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.921075 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" Jul 22 13:08:48.921222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.921092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerDied","Data":"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75"} Jul 22 13:08:48.921222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.921183 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22" event={"ID":"beba09ea-0135-4f99-8abc-1e58a150e068","Type":"ContainerDied","Data":"75208c47cd62c418d6995c16fb82ab6a59d453f12fc1fe7b3f27a6616332b2ce"} Jul 22 13:08:48.921222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.921215 2557 scope.go:117] "RemoveContainer" containerID="e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75" Jul 22 13:08:48.930393 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.930376 2557 scope.go:117] "RemoveContainer" containerID="30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb" Jul 22 13:08:48.939361 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.939339 2557 scope.go:117] "RemoveContainer" containerID="e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75" Jul 22 13:08:48.939664 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:08:48.939636 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75\": container with ID starting with e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75 not found: ID does not exist" containerID="e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75" Jul 22 13:08:48.939750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.939680 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75"} err="failed to get container status \"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75\": rpc error: code = NotFound desc = could not find container \"e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75\": container with ID starting with e2671152cf6a6b7e84731d353ab6ea2aabcaba7e1b0b074ab595a2fea4484a75 not found: ID does not exist" Jul 22 13:08:48.939750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.939724 2557 scope.go:117] "RemoveContainer" containerID="30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb" Jul 22 13:08:48.939997 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:08:48.939978 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb\": container with ID starting with 30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb not found: ID does not exist" containerID="30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb" Jul 22 13:08:48.940055 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.940002 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb"} err="failed to get container status \"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb\": rpc error: code = NotFound desc = could not find container \"30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb\": container with ID starting with 30c55dd94271d1221c50db1702413ed0c6aba4d853733f118dab8c4e02037dcb not found: ID does not exist" Jul 22 13:08:48.944492 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944464 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beba09ea-0135-4f99-8abc-1e58a150e068-istiod-ca-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944492 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944493 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-credential-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944508 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-data\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944519 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beba09ea-0135-4f99-8abc-1e58a150e068-istio-podinfo\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944532 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beba09ea-0135-4f99-8abc-1e58a150e068-istio-envoy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944544 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-istio-token\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.944639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.944556 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwtr\" (UniqueName: \"kubernetes.io/projected/beba09ea-0135-4f99-8abc-1e58a150e068-kube-api-access-7kwtr\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:08:48.953466 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.953432 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:08:48.959793 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:48.959759 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e1001375-openshift-default-55996fdcf-7rm22"] Jul 22 13:08:49.486647 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:08:49.486614 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" path="/var/lib/kubelet/pods/beba09ea-0135-4f99-8abc-1e58a150e068/volumes" Jul 22 13:09:01.985223 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.985184 2557 generic.go:356] "Generic (PLEG): container finished" podID="be235990-453b-4d99-b891-baae3ee68a88" containerID="dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0" exitCode=0 Jul 22 13:09:01.985652 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.985262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerDied","Data":"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0"} Jul 22 13:09:01.985695 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.985672 2557 scope.go:117] "RemoveContainer" containerID="dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0" Jul 22 13:09:01.986806 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.986776 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:09:01.986897 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.986863 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:09:01.986938 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:01.986908 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:09:02.915408 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:02.915358 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:02.992152 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:02.992118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerStarted","Data":"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6"} Jul 22 13:09:03.970427 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:03.970387 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:03.971614 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:03.971589 2557 patch_prober.go:28] interesting pod/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 13:09:03.971702 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:03.971646 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 13:09:04.970249 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:04.970207 2557 patch_prober.go:28] interesting pod/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 13:09:04.970700 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:04.970271 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 13:09:05.970627 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:05.970587 2557 patch_prober.go:28] interesting pod/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 13:09:05.971107 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:05.970649 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 13:09:06.970221 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:06.970173 2557 patch_prober.go:28] interesting pod/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 13:09:06.970411 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:06.970256 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 13:09:07.975637 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:07.975587 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:07.976368 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:07.976247 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:07.977424 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:07.977392 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:14.782016 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:14.781981 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:23.315111 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:23.315016 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:27.649470 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:27.649429 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:27.850454 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:27.850413 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:09:27.850699 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:27.850648 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" containerID="cri-o://16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6" gracePeriod=30 Jul 22 13:09:27.894488 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:27.894440 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:28.129973 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:28.129938 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:30.005150 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:30.005108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:30.104640 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:30.104599 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:33.107581 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.107551 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:33.145361 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.145257 2557 generic.go:356] "Generic (PLEG): container finished" podID="be235990-453b-4d99-b891-baae3ee68a88" containerID="16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6" exitCode=0 Jul 22 13:09:33.145361 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.145324 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" Jul 22 13:09:33.145568 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.145355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerDied","Data":"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6"} Jul 22 13:09:33.145568 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.145429 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d" event={"ID":"be235990-453b-4d99-b891-baae3ee68a88","Type":"ContainerDied","Data":"98c4291e50189cff69752e846dd3282978ce12e1a4392da1d9a1209281f1ede2"} Jul 22 13:09:33.145568 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.145463 2557 scope.go:117] "RemoveContainer" containerID="16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6" Jul 22 13:09:33.154360 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.154338 2557 scope.go:117] "RemoveContainer" containerID="dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0" Jul 22 13:09:33.162696 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.162675 2557 scope.go:117] "RemoveContainer" containerID="16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6" Jul 22 13:09:33.163110 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:09:33.163090 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6\": container with ID starting with 16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6 not found: ID does not exist" containerID="16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6" Jul 22 13:09:33.163173 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.163119 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6"} err="failed to get container status \"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6\": rpc error: code = NotFound desc = could not find container \"16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6\": container with ID starting with 16669fe9d7ea5012efb1fb1c4a9bc0c266e13a384c34045774cbd0ae0b473ad6 not found: ID does not exist" Jul 22 13:09:33.163173 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.163139 2557 scope.go:117] "RemoveContainer" containerID="dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0" Jul 22 13:09:33.163384 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:09:33.163365 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0\": container with ID starting with dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0 not found: ID does not exist" containerID="dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0" Jul 22 13:09:33.163437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.163391 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0"} err="failed to get container status \"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0\": rpc error: code = NotFound desc = could not find container \"dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0\": container with ID starting with dc19499633eae5e3e06e231ec9b83b1c5d5b6bddb0b97ab02e08f87b3cf860a0 not found: ID does not exist" Jul 22 13:09:33.178841 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178812 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.178978 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178861 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.178978 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178906 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.178978 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178921 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.178978 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178938 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwdfh\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.178978 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.178964 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.179228 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179066 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.179228 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179092 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.179228 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179119 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket\") pod \"be235990-453b-4d99-b891-baae3ee68a88\" (UID: \"be235990-453b-4d99-b891-baae3ee68a88\") " Jul 22 13:09:33.179396 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179238 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:09:33.179455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179415 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:09:33.179507 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179446 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:09:33.179507 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.179469 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.180305 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.180278 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:09:33.180305 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.180288 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data" (OuterVolumeSpecName: "istio-data") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:09:33.181207 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.181179 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:09:33.181207 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.181194 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:09:33.181329 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.181225 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token" (OuterVolumeSpecName: "istio-token") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:09:33.181329 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.181263 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh" (OuterVolumeSpecName: "kube-api-access-kwdfh") pod "be235990-453b-4d99-b891-baae3ee68a88" (UID: "be235990-453b-4d99-b891-baae3ee68a88"). InnerVolumeSpecName "kube-api-access-kwdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:09:33.280810 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280771 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-workload-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.280810 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280802 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be235990-453b-4d99-b891-baae3ee68a88-istiod-ca-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.280810 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280811 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-credential-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.280810 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280824 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-istio-token\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.281130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280834 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be235990-453b-4d99-b891-baae3ee68a88-istio-podinfo\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.281130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280841 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-data\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.281130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280850 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdfh\" (UniqueName: \"kubernetes.io/projected/be235990-453b-4d99-b891-baae3ee68a88-kube-api-access-kwdfh\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.281130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.280859 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be235990-453b-4d99-b891-baae3ee68a88-istio-envoy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:09:33.474750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.474689 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:09:33.475383 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.475217 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-879879e7-gw-openshift-default-6dcbf59c99-9n25d"] Jul 22 13:09:33.485633 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.485596 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be235990-453b-4d99-b891-baae3ee68a88" path="/var/lib/kubelet/pods/be235990-453b-4d99-b891-baae3ee68a88/volumes" Jul 22 13:09:33.786233 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.786147 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:33.903610 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:33.903575 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:35.457762 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:35.457701 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:35.560076 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:35.560042 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:09:59.761116 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761075 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z"] Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761412 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761422 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761433 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761439 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761446 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761452 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761467 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761472 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761524 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761531 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.761536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.761539 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="beba09ea-0135-4f99-8abc-1e58a150e068" containerName="istio-proxy" Jul 22 13:09:59.764171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.764155 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:09:59.766851 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.766826 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:09:59.767012 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.766993 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:09:59.772994 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.772968 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z"] Jul 22 13:09:59.823880 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.823844 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z\" (UID: \"b06b46ee-8333-4789-b88f-480ccc062d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:09:59.924455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.924422 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z\" (UID: \"b06b46ee-8333-4789-b88f-480ccc062d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:09:59.925075 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:09:59.925056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z\" (UID: \"b06b46ee-8333-4789-b88f-480ccc062d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:10:00.076140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:00.076049 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:10:00.220793 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:00.220548 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z"] Jul 22 13:10:00.224916 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:10:00.224885 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06b46ee_8333_4789_b88f_480ccc062d4c.slice/crio-ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145 WatchSource:0}: Error finding container ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145: Status 404 returned error can't find the container with id ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145 Jul 22 13:10:00.226881 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:00.226863 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:10:00.266936 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:00.266895 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" event={"ID":"b06b46ee-8333-4789-b88f-480ccc062d4c","Type":"ContainerStarted","Data":"ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145"} Jul 22 13:10:03.282136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:03.282042 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" event={"ID":"b06b46ee-8333-4789-b88f-480ccc062d4c","Type":"ContainerStarted","Data":"bba20b0c41f2c35d3a529674d8b6bc69c276815c650ea00850bbd71305889d62"} Jul 22 13:10:04.288163 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:04.288124 2557 generic.go:356] "Generic (PLEG): container finished" podID="b06b46ee-8333-4789-b88f-480ccc062d4c" containerID="bba20b0c41f2c35d3a529674d8b6bc69c276815c650ea00850bbd71305889d62" exitCode=0 Jul 22 13:10:04.288583 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:04.288196 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" event={"ID":"b06b46ee-8333-4789-b88f-480ccc062d4c","Type":"ContainerDied","Data":"bba20b0c41f2c35d3a529674d8b6bc69c276815c650ea00850bbd71305889d62"} Jul 22 13:10:05.421382 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:05.421356 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:10:05.478591 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:05.478550 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca\") pod \"b06b46ee-8333-4789-b88f-480ccc062d4c\" (UID: \"b06b46ee-8333-4789-b88f-480ccc062d4c\") " Jul 22 13:10:05.478928 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:05.478902 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b06b46ee-8333-4789-b88f-480ccc062d4c" (UID: "b06b46ee-8333-4789-b88f-480ccc062d4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:10:05.579889 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:05.579750 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b06b46ee-8333-4789-b88f-480ccc062d4c-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:10:06.298936 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:06.298902 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" Jul 22 13:10:06.298936 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:06.298926 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z" event={"ID":"b06b46ee-8333-4789-b88f-480ccc062d4c","Type":"ContainerDied","Data":"ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145"} Jul 22 13:10:06.299153 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:06.298971 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffdad7e74f43326ecbf09b4f1200c82132aa0e47aa8e0940c79ae338efc0a145" Jul 22 13:10:10.819077 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:10.819044 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:14.868340 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:14.868306 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:14.873000 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:14.872969 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z"] Jul 22 13:10:14.878554 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:14.878528 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e0df68f3-pwz8z"] Jul 22 13:10:15.381178 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:15.381144 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:15.479200 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:15.479167 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:15.488634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:15.488593 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b06b46ee-8333-4789-b88f-480ccc062d4c" path="/var/lib/kubelet/pods/b06b46ee-8333-4789-b88f-480ccc062d4c/volumes" Jul 22 13:10:16.180116 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:16.180083 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:16.500023 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:16.499990 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:16.675791 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:16.675755 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:17.097546 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:17.097512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:17.210347 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:17.210310 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:20.020084 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:20.020047 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:20.123042 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:20.122987 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:20.207355 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:20.207323 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:20.450582 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:20.450547 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:20.923389 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:20.923290 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:21.025187 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:21.025152 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:10:44.978120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.978087 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp"] Jul 22 13:10:44.978590 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.978446 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b06b46ee-8333-4789-b88f-480ccc062d4c" containerName="revoke-keys" Jul 22 13:10:44.978590 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.978456 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b06b46ee-8333-4789-b88f-480ccc062d4c" containerName="revoke-keys" Jul 22 13:10:44.978590 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.978505 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b06b46ee-8333-4789-b88f-480ccc062d4c" containerName="revoke-keys" Jul 22 13:10:44.978590 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.978514 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="be235990-453b-4d99-b891-baae3ee68a88" containerName="istio-proxy" Jul 22 13:10:44.982371 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.982347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:44.985028 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.985005 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:10:44.985128 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.985017 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:10:44.990296 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:44.990269 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp"] Jul 22 13:10:45.043000 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.042961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8399da33-jjqhp\" (UID: \"a2e3cbd1-16d1-46b4-a7e4-434c72b47119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:45.144448 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.144405 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8399da33-jjqhp\" (UID: \"a2e3cbd1-16d1-46b4-a7e4-434c72b47119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:45.145093 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.145069 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8399da33-jjqhp\" (UID: \"a2e3cbd1-16d1-46b4-a7e4-434c72b47119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:45.294544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.294460 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:45.445169 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.445135 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp"] Jul 22 13:10:45.450391 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:10:45.450365 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e3cbd1_16d1_46b4_a7e4_434c72b47119.slice/crio-094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6 WatchSource:0}: Error finding container 094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6: Status 404 returned error can't find the container with id 094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6 Jul 22 13:10:45.498686 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:45.498654 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" event={"ID":"a2e3cbd1-16d1-46b4-a7e4-434c72b47119","Type":"ContainerStarted","Data":"094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6"} Jul 22 13:10:46.504004 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:46.503969 2557 generic.go:356] "Generic (PLEG): container finished" podID="a2e3cbd1-16d1-46b4-a7e4-434c72b47119" containerID="9d141fc618c44d6c82dfc1cca98468d21d449add9345711df5cd58e0ad06a5d4" exitCode=0 Jul 22 13:10:46.504395 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:46.504041 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" event={"ID":"a2e3cbd1-16d1-46b4-a7e4-434c72b47119","Type":"ContainerDied","Data":"9d141fc618c44d6c82dfc1cca98468d21d449add9345711df5cd58e0ad06a5d4"} Jul 22 13:10:47.645504 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:47.645476 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:47.768462 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:47.768356 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca\") pod \"a2e3cbd1-16d1-46b4-a7e4-434c72b47119\" (UID: \"a2e3cbd1-16d1-46b4-a7e4-434c72b47119\") " Jul 22 13:10:47.768776 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:47.768751 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2e3cbd1-16d1-46b4-a7e4-434c72b47119" (UID: "a2e3cbd1-16d1-46b4-a7e4-434c72b47119"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:10:47.869584 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:47.869539 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e3cbd1-16d1-46b4-a7e4-434c72b47119-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:10:48.514866 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:48.514828 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" event={"ID":"a2e3cbd1-16d1-46b4-a7e4-434c72b47119","Type":"ContainerDied","Data":"094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6"} Jul 22 13:10:48.515044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:48.514886 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094474b6534e6d6788dbe6c12bd0bbcd784a391c503659255ab7f6a3baaf43b6" Jul 22 13:10:48.515044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:48.514838 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp" Jul 22 13:10:55.617968 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:10:55.617888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:00.074206 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:00.074100 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:00.074919 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:00.074898 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp"] Jul 22 13:11:00.077353 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:00.077328 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8399da33-jjqhp"] Jul 22 13:11:00.248168 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:00.248138 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:00.525168 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:00.525134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:01.146826 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:01.146796 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:01.275008 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:01.274973 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:01.386147 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:01.386115 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:01.486274 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:01.486246 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e3cbd1-16d1-46b4-a7e4-434c72b47119" path="/var/lib/kubelet/pods/a2e3cbd1-16d1-46b4-a7e4-434c72b47119/volumes" Jul 22 13:11:03.018592 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:03.018555 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:09.683064 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:09.683003 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:09.787415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:09.787380 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:09.885453 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:09.885400 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:10.314142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:10.314101 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:36.333288 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:36.333255 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:38.136663 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:38.136619 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:38.194225 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:38.194195 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:43.936643 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:43.936603 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:44.056233 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:44.056198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:44.177033 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:44.177002 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:53.462169 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:53.462131 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:11:53.521749 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:11:53.521689 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:00.176722 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:00.176615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:06.390101 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:06.390062 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:06.494521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:06.494489 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:06.597667 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:06.597633 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:17.159641 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:17.159602 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:17.997921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:17.997888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:25.389645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:25.389550 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:25.498697 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:25.498645 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:25.601921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:25.601884 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:25.707157 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:25.707122 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:26.531885 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:26.531846 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:26.638932 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:26.638900 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:28.447397 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:28.447363 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:34.564964 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:34.564922 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:34.671518 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:34.671487 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:35.270536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:35.270500 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:42.412316 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:42.412267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:42.510972 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:42.510933 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:42.667115 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:42.667028 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:43.018321 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:43.018278 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:43.418432 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:43.418335 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:46.529785 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:46.529744 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:46.629918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:46.629877 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:46.710413 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:46.710378 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:46.784433 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:46.784338 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:47.262129 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:47.262094 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:47.391941 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:47.391904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:50.962166 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:50.962117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:51.689703 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:51.689671 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:51.788195 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:51.788156 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:52.336017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:52.335967 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:58.420233 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.420197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:58.519206 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.519169 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:12:58.819584 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.819538 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd"] Jul 22 13:12:58.819997 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.819978 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e3cbd1-16d1-46b4-a7e4-434c72b47119" containerName="revoke-keys" Jul 22 13:12:58.820094 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.819999 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e3cbd1-16d1-46b4-a7e4-434c72b47119" containerName="revoke-keys" Jul 22 13:12:58.820094 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.820083 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e3cbd1-16d1-46b4-a7e4-434c72b47119" containerName="revoke-keys" Jul 22 13:12:58.823198 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.823171 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:12:58.825877 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.825854 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:12:58.826758 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.826734 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:12:58.833502 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.833474 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd"] Jul 22 13:12:58.903215 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:58.903176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd\" (UID: \"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:12:59.004101 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:59.004064 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd\" (UID: \"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:12:59.004756 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:59.004701 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd\" (UID: \"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:12:59.134500 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:59.134414 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:12:59.282446 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:12:59.282387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd"] Jul 22 13:12:59.284774 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:12:59.284735 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2c8676_dab5_4d1d_bfb7_e9796a1cde8e.slice/crio-c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16 WatchSource:0}: Error finding container c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16: Status 404 returned error can't find the container with id c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16 Jul 22 13:13:00.142567 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:00.142531 2557 generic.go:356] "Generic (PLEG): container finished" podID="1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" containerID="a29846f06c722abd2098dfff18311c7675407d1bcfcfa7fb20ef03e8163b3447" exitCode=0 Jul 22 13:13:00.142980 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:00.142616 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" event={"ID":"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e","Type":"ContainerDied","Data":"a29846f06c722abd2098dfff18311c7675407d1bcfcfa7fb20ef03e8163b3447"} Jul 22 13:13:00.142980 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:00.142674 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" event={"ID":"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e","Type":"ContainerStarted","Data":"c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16"} Jul 22 13:13:01.284917 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:01.284889 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:13:01.427185 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:01.427093 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca\") pod \"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e\" (UID: \"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e\") " Jul 22 13:13:01.427467 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:01.427441 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" (UID: "1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:13:01.528552 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:01.528508 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:13:02.152661 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:02.152626 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" Jul 22 13:13:02.152855 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:02.152628 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd" event={"ID":"1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e","Type":"ContainerDied","Data":"c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16"} Jul 22 13:13:02.152855 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:02.152768 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c71c1cfaa9216d947da8ddcfbcd0b9d0ba51eb86f7701fb22fda05613b1a7c16" Jul 22 13:13:09.373989 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:09.373953 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:13.904660 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:13.904627 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd"] Jul 22 13:13:13.908655 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:13.908625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:13.908935 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:13.908918 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79905e7d-nhfkd"] Jul 22 13:13:14.329370 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:14.329340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:15.488193 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:15.488160 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" path="/var/lib/kubelet/pods/1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e/volumes" Jul 22 13:13:20.522282 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:20.522243 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:20.599260 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:20.599229 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:20.829062 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:20.828967 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:21.065638 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:21.065603 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:21.206632 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:21.206583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:21.301675 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:21.301640 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:21.437989 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:21.437955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:27.831227 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:27.831161 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:27.900843 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:27.900816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:28.005984 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:28.005948 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:28.198805 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:28.198775 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:34.371975 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:34.371941 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:34.446635 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:34.446601 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:41.396078 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:41.396045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:41.740747 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:41.740701 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:48.918760 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:48.918725 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:49.366931 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:49.366898 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:52.663137 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.663055 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:13:52.663521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.663377 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" containerName="revoke-keys" Jul 22 13:13:52.663521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.663388 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" containerName="revoke-keys" Jul 22 13:13:52.663521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.663449 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2c8676-dab5-4d1d-bfb7-e9796a1cde8e" containerName="revoke-keys" Jul 22 13:13:52.666336 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.666318 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.670169 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.670139 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7caca13d\"/\"default-dockercfg-srv4h\"" Jul 22 13:13:52.670327 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.670175 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7caca13d\"/\"openshift-service-ca.crt\"" Jul 22 13:13:52.670327 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.670187 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7caca13d\"/\"e2e-lifecycle-model-7caca13d-kserve-self-signed-certs\"" Jul 22 13:13:52.670327 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.670273 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7caca13d\"/\"kube-root-ca.crt\"" Jul 22 13:13:52.675506 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.675485 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:13:52.783670 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.783670 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.783953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783778 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.783953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783841 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.783953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.783953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.783914 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x65d\" (UniqueName: \"kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885396 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885396 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885399 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885656 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885656 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885485 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885656 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885615 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885934 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x65d\" (UniqueName: \"kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.885934 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.886034 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885961 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.886034 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.885997 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.887863 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.887842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.887965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.887842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.895225 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.895204 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x65d\" (UniqueName: \"kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d\") pod \"e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:52.976123 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:52.976083 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:13:53.056120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:53.056089 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:13:53.114788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:53.114754 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:13:53.116663 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:13:53.116637 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2668beb_0784_43d9_b332_c6161e08a09a.slice/crio-543d612b4fd9b5e16960727963186a2771c4bc513a8de18fffafa73de3fa8fbc WatchSource:0}: Error finding container 543d612b4fd9b5e16960727963186a2771c4bc513a8de18fffafa73de3fa8fbc: Status 404 returned error can't find the container with id 543d612b4fd9b5e16960727963186a2771c4bc513a8de18fffafa73de3fa8fbc Jul 22 13:13:53.393973 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:53.393880 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" event={"ID":"c2668beb-0784-43d9-b332-c6161e08a09a","Type":"ContainerStarted","Data":"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e"} Jul 22 13:13:53.393973 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:53.393922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" event={"ID":"c2668beb-0784-43d9-b332-c6161e08a09a","Type":"ContainerStarted","Data":"543d612b4fd9b5e16960727963186a2771c4bc513a8de18fffafa73de3fa8fbc"} Jul 22 13:13:53.411869 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:13:53.411807 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" podStartSLOduration=1.41178662 podStartE2EDuration="1.41178662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:13:53.411258051 +0000 UTC m=+1972.498747267" watchObservedRunningTime="2026-07-22 13:13:53.41178662 +0000 UTC m=+1972.499275802" Jul 22 13:14:02.976245 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:02.976204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:14:02.978689 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:02.976261 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:14:02.988998 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:02.988973 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:14:03.456803 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:03.456777 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:14:17.630387 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:17.630346 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:33.350500 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:33.350459 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:37.931385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:37.931351 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:40.300008 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:40.299973 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:45.278521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:45.278480 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:14:45.374924 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:45.374890 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:49.876993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:49.876955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:14:51.924039 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:14:51.924005 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:00.141965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.141870 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf"] Jul 22 13:15:00.145286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.145263 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" Jul 22 13:15:00.148149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.148123 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:15:00.154076 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.154047 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf"] Jul 22 13:15:00.157125 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.157100 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" Jul 22 13:15:00.299191 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.299141 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" podUID="c2668beb-0784-43d9-b332-c6161e08a09a" containerName="main" containerID="cri-o://71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e" gracePeriod=45 Jul 22 13:15:00.304775 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.304745 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf"] Jul 22 13:15:00.307074 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:15:00.307044 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fc0dfe_d6ae_4e9b_8cdf_2be7d135fd4e.slice/crio-592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582 WatchSource:0}: Error finding container 592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582: Status 404 returned error can't find the container with id 592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582 Jul 22 13:15:00.551564 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.551528 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:15:00.603533 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603502 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.603750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603548 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.603750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603571 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x65d\" (UniqueName: \"kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.603750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603598 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.603750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603666 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.603750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603736 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm\") pod \"c2668beb-0784-43d9-b332-c6161e08a09a\" (UID: \"c2668beb-0784-43d9-b332-c6161e08a09a\") " Jul 22 13:15:00.604046 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.603949 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:00.604108 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.604064 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home" (OuterVolumeSpecName: "home") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:00.604165 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.604135 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.604224 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.604187 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:00.606270 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.606234 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d" (OuterVolumeSpecName: "kube-api-access-7x65d") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "kube-api-access-7x65d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:15:00.606270 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.606241 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm" (OuterVolumeSpecName: "dshm") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:00.606270 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.606241 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2668beb-0784-43d9-b332-c6161e08a09a" (UID: "c2668beb-0784-43d9-b332-c6161e08a09a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:15:00.704723 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.704686 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2668beb-0784-43d9-b332-c6161e08a09a-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.704929 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.704749 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x65d\" (UniqueName: \"kubernetes.io/projected/c2668beb-0784-43d9-b332-c6161e08a09a-kube-api-access-7x65d\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.704929 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.704765 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.704929 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.704776 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.704929 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.704786 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2668beb-0784-43d9-b332-c6161e08a09a-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:15:00.705927 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.705894 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" event={"ID":"84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e","Type":"ContainerStarted","Data":"864a5e1c31fd47b7f6bb3a8aa4bc3602733ca07de19391e53f89ba156fd59dc1"} Jul 22 13:15:00.706073 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.705958 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" event={"ID":"84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e","Type":"ContainerStarted","Data":"592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582"} Jul 22 13:15:00.707247 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.707219 2557 generic.go:356] "Generic (PLEG): container finished" podID="c2668beb-0784-43d9-b332-c6161e08a09a" containerID="71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e" exitCode=0 Jul 22 13:15:00.707385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.707266 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" Jul 22 13:15:00.707385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.707308 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" event={"ID":"c2668beb-0784-43d9-b332-c6161e08a09a","Type":"ContainerDied","Data":"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e"} Jul 22 13:15:00.707385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.707362 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk" event={"ID":"c2668beb-0784-43d9-b332-c6161e08a09a","Type":"ContainerDied","Data":"543d612b4fd9b5e16960727963186a2771c4bc513a8de18fffafa73de3fa8fbc"} Jul 22 13:15:00.707548 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.707386 2557 scope.go:117] "RemoveContainer" containerID="71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e" Jul 22 13:15:00.717276 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.717258 2557 scope.go:117] "RemoveContainer" containerID="71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e" Jul 22 13:15:00.717634 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:15:00.717609 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e\": container with ID starting with 71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e not found: ID does not exist" containerID="71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e" Jul 22 13:15:00.717817 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.717642 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e"} err="failed to get container status \"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e\": rpc error: code = NotFound desc = could not find container \"71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e\": container with ID starting with 71b1a68cd442122a4a15b3e4635a09c493bad8bc9b136677a40ab77ed5d0f91e not found: ID does not exist" Jul 22 13:15:00.724879 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.724821 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" podStartSLOduration=0.724804027 podStartE2EDuration="724.804027ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:15:00.723606606 +0000 UTC m=+2039.811095788" watchObservedRunningTime="2026-07-22 13:15:00.724804027 +0000 UTC m=+2039.812293208" Jul 22 13:15:00.738201 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.738164 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:15:00.745019 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:00.744965 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7caca13d/e2e-lifecycle-model-7caca13d-kserve-6b74f79cc5-7xfrk"] Jul 22 13:15:01.487385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:01.487341 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2668beb-0784-43d9-b332-c6161e08a09a" path="/var/lib/kubelet/pods/c2668beb-0784-43d9-b332-c6161e08a09a/volumes" Jul 22 13:15:01.713652 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:01.713606 2557 generic.go:356] "Generic (PLEG): container finished" podID="84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e" containerID="864a5e1c31fd47b7f6bb3a8aa4bc3602733ca07de19391e53f89ba156fd59dc1" exitCode=0 Jul 22 13:15:01.713859 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:01.713721 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" event={"ID":"84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e","Type":"ContainerDied","Data":"864a5e1c31fd47b7f6bb3a8aa4bc3602733ca07de19391e53f89ba156fd59dc1"} Jul 22 13:15:02.865614 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:02.865570 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" Jul 22 13:15:03.726142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:03.726107 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" Jul 22 13:15:03.726142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:03.726118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745435-j6cxf" event={"ID":"84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e","Type":"ContainerDied","Data":"592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582"} Jul 22 13:15:03.726377 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:03.726164 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592491b36673b6225cb2c35eea934a975bd23028a271cd387b07f56014f2a582" Jul 22 13:15:16.352481 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:16.352446 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:16.448522 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:16.448488 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:22.999522 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:22.999435 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:23.296917 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:23.296829 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:29.255306 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:29.255259 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:35.028342 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:35.028296 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:35.103666 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:35.103628 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:35.437880 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:35.437787 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:35.787024 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:35.786987 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:36.752135 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:36.752099 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:36.853362 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:36.853314 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:36.951670 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:36.951630 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:37.782451 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:37.782416 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:37.895503 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:37.895450 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:39.444150 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:39.444107 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:45.521171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:45.521138 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:45.622899 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:45.622867 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:46.351439 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:46.351400 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:52.798885 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:52.798844 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:53.647702 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:53.647634 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:15:53.746448 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:15:53.746398 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:03.570412 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:03.570379 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:03.923523 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:03.923418 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:24.989620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:24.989584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:32.042498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:32.042460 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:32.141405 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:32.141375 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:32.414111 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:32.414024 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:32.781625 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:32.781592 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:32.981600 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:32.981559 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:35.826070 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:35.826009 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:35.945036 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:35.944986 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:36.023965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:36.023930 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:36.136278 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:36.136182 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:36.679840 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:36.679793 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:36.785613 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:36.785584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:40.290610 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.290572 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:40.629884 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.629788 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:40.932910 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.932834 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:16:40.933165 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933154 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e" containerName="cleanup" Jul 22 13:16:40.933243 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933166 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e" containerName="cleanup" Jul 22 13:16:40.933243 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933181 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2668beb-0784-43d9-b332-c6161e08a09a" containerName="main" Jul 22 13:16:40.933243 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933187 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2668beb-0784-43d9-b332-c6161e08a09a" containerName="main" Jul 22 13:16:40.933335 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933255 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2668beb-0784-43d9-b332-c6161e08a09a" containerName="main" Jul 22 13:16:40.933335 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.933264 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e" containerName="cleanup" Jul 22 13:16:40.942373 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.942270 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:40.945263 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.945237 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ab0ebe-openshift-default-dockercfg-kb6qh\"" Jul 22 13:16:40.951406 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.951386 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:16:40.969629 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:40.969603 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:41.037523 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037488 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037523 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037525 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5p4\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037762 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.037813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.038130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.037818 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.138971 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.138942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5p4\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139151 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139151 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139151 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139151 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139121 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139151 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139145 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139438 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139438 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139438 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139438 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139749 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139696 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.139969 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.140051 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.139968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.140173 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.140149 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.142199 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.142174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.142387 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.142368 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.148171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.148145 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5p4\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.148453 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.148435 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token\") pod \"e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.210270 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.210241 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:41.254369 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.254341 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:41.389059 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.389034 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:16:41.391179 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:16:41.391149 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462d18a3_1bbf_4043_b39b_7437cfd1bc29.slice/crio-85d38ddbc4f3268ead36280254ccbd2cfb216255b723ae8195d91e8f3c19fe43 WatchSource:0}: Error finding container 85d38ddbc4f3268ead36280254ccbd2cfb216255b723ae8195d91e8f3c19fe43: Status 404 returned error can't find the container with id 85d38ddbc4f3268ead36280254ccbd2cfb216255b723ae8195d91e8f3c19fe43 Jul 22 13:16:41.393211 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.393178 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:16:41.393299 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.393247 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:16:41.393299 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:41.393279 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:16:42.193559 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:42.193523 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerStarted","Data":"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad"} Jul 22 13:16:42.193739 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:42.193578 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerStarted","Data":"85d38ddbc4f3268ead36280254ccbd2cfb216255b723ae8195d91e8f3c19fe43"} Jul 22 13:16:42.219153 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:42.219102 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podStartSLOduration=2.219086716 podStartE2EDuration="2.219086716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:16:42.217849531 +0000 UTC m=+2141.305338704" watchObservedRunningTime="2026-07-22 13:16:42.219086716 +0000 UTC m=+2141.306575896" Jul 22 13:16:42.254762 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:42.254741 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:42.259499 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:42.259471 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:43.197474 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:43.197446 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:43.198224 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:43.198205 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:16:44.254153 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:44.254118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:44.344361 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:44.344328 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:44.521766 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:44.521664 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:45.045633 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:45.045602 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:45.146529 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:45.146496 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:45.245885 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:45.245850 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:48.804846 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:48.804816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:57.281139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:57.281094 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:57.543054 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:57.542970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:57.640995 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:57.640966 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:16:59.239879 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:59.239835 2557 patch_prober.go:28] interesting pod/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 13:16:59.240240 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:16:59.239893 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 13:17:00.282126 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.282091 2557 generic.go:356] "Generic (PLEG): container finished" podID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerID="0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad" exitCode=0 Jul 22 13:17:00.282645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.282148 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerDied","Data":"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad"} Jul 22 13:17:00.282739 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.282653 2557 scope.go:117] "RemoveContainer" containerID="0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad" Jul 22 13:17:00.283673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.283649 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:17:00.283758 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.283743 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:17:00.283803 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:00.283791 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:17:01.288875 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:01.288840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerStarted","Data":"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6"} Jul 22 13:17:01.617657 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:01.617555 2557 scope.go:117] "RemoveContainer" containerID="9d141fc618c44d6c82dfc1cca98468d21d449add9345711df5cd58e0ad06a5d4" Jul 22 13:17:01.628519 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:01.628496 2557 scope.go:117] "RemoveContainer" containerID="bba20b0c41f2c35d3a529674d8b6bc69c276815c650ea00850bbd71305889d62" Jul 22 13:17:02.254687 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:02.254648 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:17:02.254942 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:02.254918 2557 patch_prober.go:28] interesting pod/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 13:17:02.255027 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:02.254993 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 13:17:03.255142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:03.255098 2557 patch_prober.go:28] interesting pod/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 13:17:03.255741 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:03.255177 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 13:17:04.255530 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:04.255487 2557 patch_prober.go:28] interesting pod/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 13:17:04.256048 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:04.255550 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 13:17:05.258830 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:05.258799 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:17:05.259302 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:05.259276 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:17:05.260108 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:05.260088 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:17:06.208220 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:06.208185 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:17:09.192624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:09.192577 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:17:17.562600 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.562550 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:17:17.565254 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.565231 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.568145 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.568116 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab0ebe\"/\"default-dockercfg-hkpxb\"" Jul 22 13:17:17.569100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.569071 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab0ebe\"/\"openshift-service-ca.crt\"" Jul 22 13:17:17.569236 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.569193 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab0ebe\"/\"test-model-ab0ebe-kserve-self-signed-certs\"" Jul 22 13:17:17.569236 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.569214 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab0ebe\"/\"kube-root-ca.crt\"" Jul 22 13:17:17.575946 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.575904 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:17:17.600351 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.600314 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:17:17.670398 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670344 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.670620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670421 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.670620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.670620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670476 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.670620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670496 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgzd\" (UniqueName: \"kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.670620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.670598 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.771906 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.771859 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.771906 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.771914 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.771936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.771963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgzd\" (UniqueName: \"kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.772022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772142 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.772078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772396 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.772362 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772396 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.772391 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.772587 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.772460 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.774386 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.774357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.774572 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.774553 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.781536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.781491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgzd\" (UniqueName: \"kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd\") pod \"test-model-ab0ebe-kserve-6c987ddd59-d5s9w\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:17.876772 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:17.876640 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:18.022580 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:18.022549 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:17:18.024680 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:17:18.024654 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d15316_cf50_4009_bef5_d923cdee5150.slice/crio-d876e8eeb0a0b278fe2e7cbc4cf40f10a7e725555a029cbe116fab8e2a952b4b WatchSource:0}: Error finding container d876e8eeb0a0b278fe2e7cbc4cf40f10a7e725555a029cbe116fab8e2a952b4b: Status 404 returned error can't find the container with id d876e8eeb0a0b278fe2e7cbc4cf40f10a7e725555a029cbe116fab8e2a952b4b Jul 22 13:17:18.369587 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:18.369545 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" event={"ID":"44d15316-cf50-4009-bef5-d923cdee5150","Type":"ContainerStarted","Data":"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9"} Jul 22 13:17:18.369587 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:18.369593 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" event={"ID":"44d15316-cf50-4009-bef5-d923cdee5150","Type":"ContainerStarted","Data":"d876e8eeb0a0b278fe2e7cbc4cf40f10a7e725555a029cbe116fab8e2a952b4b"} Jul 22 13:17:18.389298 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:18.389245 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" podStartSLOduration=1.389227765 podStartE2EDuration="1.389227765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:17:18.388302311 +0000 UTC m=+2177.475791489" watchObservedRunningTime="2026-07-22 13:17:18.389227765 +0000 UTC m=+2177.476716946" Jul 22 13:17:27.877534 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:27.877490 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:27.877534 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:27.877541 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:27.889824 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:27.889797 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:28.430845 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:28.430818 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:17:58.385415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:58.385378 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:17:59.281699 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.281663 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:17:59.306283 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.306252 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:17:59.429140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.429103 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd"] Jul 22 13:17:59.432623 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.432603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:17:59.435215 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.435195 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:17:59.435215 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.435208 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:17:59.441060 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.441030 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd"] Jul 22 13:17:59.442899 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.442876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd\" (UID: \"a56b8c0d-460c-43c3-9574-1a2b8eefef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:17:59.543770 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.543671 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd\" (UID: \"a56b8c0d-460c-43c3-9574-1a2b8eefef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:17:59.544259 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.544238 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd\" (UID: \"a56b8c0d-460c-43c3-9574-1a2b8eefef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:17:59.744192 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.744164 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:17:59.875686 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:17:59.875660 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd"] Jul 22 13:17:59.877589 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:17:59.877560 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56b8c0d_460c_43c3_9574_1a2b8eefef0b.slice/crio-5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace WatchSource:0}: Error finding container 5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace: Status 404 returned error can't find the container with id 5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace Jul 22 13:18:00.058233 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:00.058149 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:00.158808 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:00.158772 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:00.562624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:00.562581 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" event={"ID":"a56b8c0d-460c-43c3-9574-1a2b8eefef0b","Type":"ContainerStarted","Data":"72defdbe47aaffdfea3a7ede4d7c60026839d4ea5794ad23158f0754d20c52b6"} Jul 22 13:18:00.563120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:00.562640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" event={"ID":"a56b8c0d-460c-43c3-9574-1a2b8eefef0b","Type":"ContainerStarted","Data":"5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace"} Jul 22 13:18:00.579868 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:00.579808 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" podStartSLOduration=1.579790512 podStartE2EDuration="1.579790512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:18:00.577500198 +0000 UTC m=+2219.664989374" watchObservedRunningTime="2026-07-22 13:18:00.579790512 +0000 UTC m=+2219.667279733" Jul 22 13:18:01.568210 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:01.568175 2557 generic.go:356] "Generic (PLEG): container finished" podID="a56b8c0d-460c-43c3-9574-1a2b8eefef0b" containerID="72defdbe47aaffdfea3a7ede4d7c60026839d4ea5794ad23158f0754d20c52b6" exitCode=0 Jul 22 13:18:01.568692 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:01.568280 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" event={"ID":"a56b8c0d-460c-43c3-9574-1a2b8eefef0b","Type":"ContainerDied","Data":"72defdbe47aaffdfea3a7ede4d7c60026839d4ea5794ad23158f0754d20c52b6"} Jul 22 13:18:02.704409 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:02.704380 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:18:02.771271 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:02.771240 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca\") pod \"a56b8c0d-460c-43c3-9574-1a2b8eefef0b\" (UID: \"a56b8c0d-460c-43c3-9574-1a2b8eefef0b\") " Jul 22 13:18:02.771635 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:02.771613 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a56b8c0d-460c-43c3-9574-1a2b8eefef0b" (UID: "a56b8c0d-460c-43c3-9574-1a2b8eefef0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:02.872993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:02.872923 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56b8c0d-460c-43c3-9574-1a2b8eefef0b-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:03.577372 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:03.577327 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" event={"ID":"a56b8c0d-460c-43c3-9574-1a2b8eefef0b","Type":"ContainerDied","Data":"5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace"} Jul 22 13:18:03.577372 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:03.577353 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd" Jul 22 13:18:03.577645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:03.577385 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2fd08bf3ab9eaf0336e97b2587198845a512a8441f199664bcaeb2d3c71ace" Jul 22 13:18:09.933471 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:09.933417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:14.301763 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.301725 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" podUID="44d15316-cf50-4009-bef5-d923cdee5150" containerName="main" containerID="cri-o://df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9" gracePeriod=45 Jul 22 13:18:14.497136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.497110 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:14.515490 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.515437 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd"] Jul 22 13:18:14.517683 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.517659 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50f07d-pnzvd"] Jul 22 13:18:14.542950 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.542932 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:18:14.577867 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.577806 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.577968 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.577873 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.577968 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.577901 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.577968 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.577930 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pgzd\" (UniqueName: \"kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.578127 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578022 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.578127 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578033 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache" (OuterVolumeSpecName: "model-cache") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:14.578127 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578071 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm\") pod \"44d15316-cf50-4009-bef5-d923cdee5150\" (UID: \"44d15316-cf50-4009-bef5-d923cdee5150\") " Jul 22 13:18:14.578277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578131 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:14.578277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578144 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home" (OuterVolumeSpecName: "home") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:14.578402 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578387 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.578456 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578408 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.578456 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.578421 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.580088 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.580066 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd" (OuterVolumeSpecName: "kube-api-access-5pgzd") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "kube-api-access-5pgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:14.580487 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.580470 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm" (OuterVolumeSpecName: "dshm") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:14.580562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.580495 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44d15316-cf50-4009-bef5-d923cdee5150" (UID: "44d15316-cf50-4009-bef5-d923cdee5150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:18:14.627634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.627608 2557 generic.go:356] "Generic (PLEG): container finished" podID="44d15316-cf50-4009-bef5-d923cdee5150" containerID="df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9" exitCode=0 Jul 22 13:18:14.627770 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.627654 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" Jul 22 13:18:14.627770 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.627660 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" event={"ID":"44d15316-cf50-4009-bef5-d923cdee5150","Type":"ContainerDied","Data":"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9"} Jul 22 13:18:14.627893 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.627770 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w" event={"ID":"44d15316-cf50-4009-bef5-d923cdee5150","Type":"ContainerDied","Data":"d876e8eeb0a0b278fe2e7cbc4cf40f10a7e725555a029cbe116fab8e2a952b4b"} Jul 22 13:18:14.627893 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.627794 2557 scope.go:117] "RemoveContainer" containerID="df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9" Jul 22 13:18:14.636247 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.636232 2557 scope.go:117] "RemoveContainer" containerID="df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9" Jul 22 13:18:14.636497 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:18:14.636478 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9\": container with ID starting with df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9 not found: ID does not exist" containerID="df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9" Jul 22 13:18:14.636559 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.636509 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9"} err="failed to get container status \"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9\": rpc error: code = NotFound desc = could not find container \"df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9\": container with ID starting with df3368399bc51d7568aafefcb231ac226c4ffd116b943fff126f3ec33937bbf9 not found: ID does not exist" Jul 22 13:18:14.648763 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.648741 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:18:14.652577 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.652554 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ab0ebe/test-model-ab0ebe-kserve-6c987ddd59-d5s9w"] Jul 22 13:18:14.679030 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.679006 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d15316-cf50-4009-bef5-d923cdee5150-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.679030 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.679027 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d15316-cf50-4009-bef5-d923cdee5150-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.679140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.679046 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgzd\" (UniqueName: \"kubernetes.io/projected/44d15316-cf50-4009-bef5-d923cdee5150-kube-api-access-5pgzd\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.922325 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:14.922259 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:15.486893 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:15.486858 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d15316-cf50-4009-bef5-d923cdee5150" path="/var/lib/kubelet/pods/44d15316-cf50-4009-bef5-d923cdee5150/volumes" Jul 22 13:18:15.487376 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:15.487323 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56b8c0d-460c-43c3-9574-1a2b8eefef0b" path="/var/lib/kubelet/pods/a56b8c0d-460c-43c3-9574-1a2b8eefef0b/volumes" Jul 22 13:18:21.294220 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.294186 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:21.356181 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.356149 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:21.574594 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.574498 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:21.807652 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.807614 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:21.931569 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931499 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4"] Jul 22 13:18:21.931837 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931823 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d15316-cf50-4009-bef5-d923cdee5150" containerName="main" Jul 22 13:18:21.931883 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931837 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d15316-cf50-4009-bef5-d923cdee5150" containerName="main" Jul 22 13:18:21.931883 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931859 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56b8c0d-460c-43c3-9574-1a2b8eefef0b" containerName="revoke-keys" Jul 22 13:18:21.931883 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931864 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56b8c0d-460c-43c3-9574-1a2b8eefef0b" containerName="revoke-keys" Jul 22 13:18:21.931976 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931916 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d15316-cf50-4009-bef5-d923cdee5150" containerName="main" Jul 22 13:18:21.931976 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.931926 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56b8c0d-460c-43c3-9574-1a2b8eefef0b" containerName="revoke-keys" Jul 22 13:18:21.935027 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.935009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:21.937455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.937436 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:18:21.937544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.937474 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:18:21.942648 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:21.942626 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4"] Jul 22 13:18:22.046741 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.046639 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4\" (UID: \"b1fdd21f-0153-4354-91e3-b072b8b2c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:22.147349 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.147320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4\" (UID: \"b1fdd21f-0153-4354-91e3-b072b8b2c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:22.147935 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.147917 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4\" (UID: \"b1fdd21f-0153-4354-91e3-b072b8b2c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:22.246398 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.246370 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:22.372048 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.372022 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4"] Jul 22 13:18:22.373964 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:18:22.373936 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fdd21f_0153_4354_91e3_b072b8b2c324.slice/crio-b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393 WatchSource:0}: Error finding container b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393: Status 404 returned error can't find the container with id b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393 Jul 22 13:18:22.667236 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:22.667200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" event={"ID":"b1fdd21f-0153-4354-91e3-b072b8b2c324","Type":"ContainerStarted","Data":"b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393"} Jul 22 13:18:23.672592 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:23.672556 2557 generic.go:356] "Generic (PLEG): container finished" podID="b1fdd21f-0153-4354-91e3-b072b8b2c324" containerID="3996b0be9d77d2c65c5253c3bf406f5fc187c8874ae1357640c5dc992bf3f564" exitCode=0 Jul 22 13:18:23.673035 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:23.672641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" event={"ID":"b1fdd21f-0153-4354-91e3-b072b8b2c324","Type":"ContainerDied","Data":"3996b0be9d77d2c65c5253c3bf406f5fc187c8874ae1357640c5dc992bf3f564"} Jul 22 13:18:24.807325 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:24.807299 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:24.870887 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:24.870863 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca\") pod \"b1fdd21f-0153-4354-91e3-b072b8b2c324\" (UID: \"b1fdd21f-0153-4354-91e3-b072b8b2c324\") " Jul 22 13:18:24.871173 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:24.871153 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1fdd21f-0153-4354-91e3-b072b8b2c324" (UID: "b1fdd21f-0153-4354-91e3-b072b8b2c324"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:24.971772 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:24.971751 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fdd21f-0153-4354-91e3-b072b8b2c324-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:25.500302 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:25.500273 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:25.556647 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:25.556626 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:25.682719 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:25.682691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" event={"ID":"b1fdd21f-0153-4354-91e3-b072b8b2c324","Type":"ContainerDied","Data":"b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393"} Jul 22 13:18:25.682828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:25.682751 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b440a215dacff8658db7889555c6eb488cd0696bccc91548e585560fd67b2393" Jul 22 13:18:25.682828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:25.682703 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4" Jul 22 13:18:32.428616 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:32.428584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:37.008831 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:37.008785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:37.012875 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:37.012844 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4"] Jul 22 13:18:37.014455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:37.014433 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0ebe-rv6v4"] Jul 22 13:18:37.408234 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:37.408155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:37.486154 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:37.486121 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fdd21f-0153-4354-91e3-b072b8b2c324" path="/var/lib/kubelet/pods/b1fdd21f-0153-4354-91e3-b072b8b2c324/volumes" Jul 22 13:18:43.478824 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:43.478789 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:43.556072 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:43.556044 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:43.764003 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:43.763915 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:18:43.764205 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:43.764177 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" containerID="cri-o://b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6" gracePeriod=30 Jul 22 13:18:43.790917 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:43.790888 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:44.021637 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:44.021542 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:44.391123 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:44.391039 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:44.725519 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:44.725484 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:44.876896 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:44.876862 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:47.389188 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:47.389151 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:48.876191 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:48.876159 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:49.030034 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.029953 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:18:49.096316 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096278 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096316 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096322 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096354 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096395 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk5p4\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096449 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096475 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096501 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096534 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096587 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token\") pod \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\" (UID: \"462d18a3-1bbf-4043-b39b-7437cfd1bc29\") " Jul 22 13:18:49.096914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096685 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:49.096914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096744 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:49.096914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.096795 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:49.097120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.097002 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-credential-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.097120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.097030 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istiod-ca-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.097120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.097049 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.097120 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.097083 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:49.097560 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.097521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data" (OuterVolumeSpecName: "istio-data") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:49.098911 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.098863 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:18:49.098911 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.098882 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:49.099168 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.099149 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token" (OuterVolumeSpecName: "istio-token") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:49.099233 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.099149 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4" (OuterVolumeSpecName: "kube-api-access-tk5p4") pod "462d18a3-1bbf-4043-b39b-7437cfd1bc29" (UID: "462d18a3-1bbf-4043-b39b-7437cfd1bc29"). InnerVolumeSpecName "kube-api-access-tk5p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:49.198030 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.197988 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-token\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.198030 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.198026 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5p4\" (UniqueName: \"kubernetes.io/projected/462d18a3-1bbf-4043-b39b-7437cfd1bc29-kube-api-access-tk5p4\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.198030 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.198038 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-envoy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.198283 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.198046 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-podinfo\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.198283 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.198055 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-workload-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.198283 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.198065 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d18a3-1bbf-4043-b39b-7437cfd1bc29-istio-data\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:18:49.796222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.796124 2557 generic.go:356] "Generic (PLEG): container finished" podID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerID="b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6" exitCode=0 Jul 22 13:18:49.796222 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.796178 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" Jul 22 13:18:49.796415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.796222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerDied","Data":"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6"} Jul 22 13:18:49.796415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.796283 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w" event={"ID":"462d18a3-1bbf-4043-b39b-7437cfd1bc29","Type":"ContainerDied","Data":"85d38ddbc4f3268ead36280254ccbd2cfb216255b723ae8195d91e8f3c19fe43"} Jul 22 13:18:49.796415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.796307 2557 scope.go:117] "RemoveContainer" containerID="b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6" Jul 22 13:18:49.804904 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.804881 2557 scope.go:117] "RemoveContainer" containerID="0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad" Jul 22 13:18:49.813313 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.813288 2557 scope.go:117] "RemoveContainer" containerID="b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6" Jul 22 13:18:49.813635 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:18:49.813614 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6\": container with ID starting with b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6 not found: ID does not exist" containerID="b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6" Jul 22 13:18:49.813820 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.813643 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6"} err="failed to get container status \"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6\": rpc error: code = NotFound desc = could not find container \"b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6\": container with ID starting with b651c3709bd262e67f64bbd562259860047e0c2cd6dbdfbf8adad741ab292da6 not found: ID does not exist" Jul 22 13:18:49.813820 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.813666 2557 scope.go:117] "RemoveContainer" containerID="0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad" Jul 22 13:18:49.814010 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:18:49.813947 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad\": container with ID starting with 0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad not found: ID does not exist" containerID="0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad" Jul 22 13:18:49.814426 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.814008 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad"} err="failed to get container status \"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad\": rpc error: code = NotFound desc = could not find container \"0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad\": container with ID starting with 0c9b3f2fe17e336a3f1eb68fd775a25cabcff7fe751f8d9d014368a6b513d9ad not found: ID does not exist" Jul 22 13:18:49.816360 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.816338 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:18:49.820426 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:49.820392 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ab0ebe-openshift-default-774997885c-szv8w"] Jul 22 13:18:51.486118 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:51.486070 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" path="/var/lib/kubelet/pods/462d18a3-1bbf-4043-b39b-7437cfd1bc29/volumes" Jul 22 13:18:51.944658 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:51.944613 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:52.460702 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.460656 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:52.808744 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.808346 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:18:52.809367 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809336 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fdd21f-0153-4354-91e3-b072b8b2c324" containerName="revoke-keys" Jul 22 13:18:52.809367 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809361 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fdd21f-0153-4354-91e3-b072b8b2c324" containerName="revoke-keys" Jul 22 13:18:52.809544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809376 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:18:52.809544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809384 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:18:52.809544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809480 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:18:52.809544 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.809493 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fdd21f-0153-4354-91e3-b072b8b2c324" containerName="revoke-keys" Jul 22 13:18:52.812554 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.812535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.816822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.816794 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5886b7-openshift-default-dockercfg-nxnf9\"" Jul 22 13:18:52.827171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.827141 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:18:52.834447 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834421 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrt5n\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834464 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834502 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834527 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834815 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834815 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834634 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834815 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.834815 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.834758 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.881280 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.881246 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:52.935418 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935379 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935418 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935433 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935495 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935585 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.935673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrt5n\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935861 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936044 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.935930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936205 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.936104 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936276 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.936206 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.936593 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.936570 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.938143 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.938124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.938236 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.938160 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.943556 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.943533 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:52.943872 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:52.943853 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrt5n\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n\") pod \"e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:53.122990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.122903 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:53.143420 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.143394 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:53.262025 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.261999 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:18:53.264031 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:18:53.264002 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15469df8_a467_4419_a934_5d4907bc16c5.slice/crio-5c54649d283dc851e76328068c482391ec2506546716b5efc5cb26cd85ddfecc WatchSource:0}: Error finding container 5c54649d283dc851e76328068c482391ec2506546716b5efc5cb26cd85ddfecc: Status 404 returned error can't find the container with id 5c54649d283dc851e76328068c482391ec2506546716b5efc5cb26cd85ddfecc Jul 22 13:18:53.266124 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.266095 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:18:53.266216 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.266158 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:18:53.266216 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.266187 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:18:53.819147 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.819109 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerStarted","Data":"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f"} Jul 22 13:18:53.819542 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.819163 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerStarted","Data":"5c54649d283dc851e76328068c482391ec2506546716b5efc5cb26cd85ddfecc"} Jul 22 13:18:53.844174 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:53.844121 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podStartSLOduration=1.844103413 podStartE2EDuration="1.844103413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:18:53.843033772 +0000 UTC m=+2272.930522965" watchObservedRunningTime="2026-07-22 13:18:53.844103413 +0000 UTC m=+2272.931592593" Jul 22 13:18:54.123947 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:54.123863 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:54.129016 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:54.128991 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:54.824002 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:54.823973 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:54.824859 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:54.824836 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:18:55.849866 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:55.849818 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:55.985188 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:55.985139 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:56.226055 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:56.226001 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:56.326343 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:56.326310 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:56.837094 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:56.837061 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:18:56.936297 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:18:56.936265 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:00.345300 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:00.345270 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:01.661683 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:01.661646 2557 scope.go:117] "RemoveContainer" containerID="a29846f06c722abd2098dfff18311c7675407d1bcfcfa7fb20ef03e8163b3447" Jul 22 13:19:08.655035 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.654939 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx"] Jul 22 13:19:08.655411 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.655273 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:19:08.655411 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.655284 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:19:08.655411 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.655339 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="462d18a3-1bbf-4043-b39b-7437cfd1bc29" containerName="istio-proxy" Jul 22 13:19:08.657335 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.657317 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:08.659780 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.659760 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:19:08.659899 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.659873 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:19:08.666773 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.666751 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx"] Jul 22 13:19:08.782534 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.782500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx\" (UID: \"3e4e645c-03f4-4b57-853f-dbf65292429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:08.883931 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.883898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx\" (UID: \"3e4e645c-03f4-4b57-853f-dbf65292429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:08.884520 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.884502 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx\" (UID: \"3e4e645c-03f4-4b57-853f-dbf65292429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:08.889588 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.889563 2557 generic.go:356] "Generic (PLEG): container finished" podID="15469df8-a467-4419-a934-5d4907bc16c5" containerID="4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f" exitCode=0 Jul 22 13:19:08.889732 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.889631 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerDied","Data":"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f"} Jul 22 13:19:08.890006 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.889993 2557 scope.go:117] "RemoveContainer" containerID="4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f" Jul 22 13:19:08.890929 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.890904 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:19:08.890979 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.890966 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:19:08.891024 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.890997 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 13:19:08.968525 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:08.968500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:09.120289 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.120061 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx"] Jul 22 13:19:09.123306 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:19:09.123275 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4e645c_03f4_4b57_853f_dbf65292429d.slice/crio-fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316 WatchSource:0}: Error finding container fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316: Status 404 returned error can't find the container with id fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316 Jul 22 13:19:09.227558 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.227524 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:09.328997 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.328959 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:09.824214 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.824168 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:19:09.911382 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.911336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerStarted","Data":"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4"} Jul 22 13:19:09.917202 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.917159 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" event={"ID":"3e4e645c-03f4-4b57-853f-dbf65292429d","Type":"ContainerStarted","Data":"2e11f1f1222502ee61be19f595daa72544a75a6527f9d02aa032806f91a48076"} Jul 22 13:19:09.917384 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.917235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" event={"ID":"3e4e645c-03f4-4b57-853f-dbf65292429d","Type":"ContainerStarted","Data":"fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316"} Jul 22 13:19:09.947835 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:09.947775 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" podStartSLOduration=1.947757169 podStartE2EDuration="1.947757169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:19:09.946958782 +0000 UTC m=+2289.034447980" watchObservedRunningTime="2026-07-22 13:19:09.947757169 +0000 UTC m=+2289.035246349" Jul 22 13:19:10.124965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:10.123783 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:19:10.125150 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:10.125006 2557 patch_prober.go:28] interesting pod/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 13:19:10.125150 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:10.125093 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 13:19:10.924247 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:10.924215 2557 generic.go:356] "Generic (PLEG): container finished" podID="3e4e645c-03f4-4b57-853f-dbf65292429d" containerID="2e11f1f1222502ee61be19f595daa72544a75a6527f9d02aa032806f91a48076" exitCode=0 Jul 22 13:19:10.924723 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:10.924306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" event={"ID":"3e4e645c-03f4-4b57-853f-dbf65292429d","Type":"ContainerDied","Data":"2e11f1f1222502ee61be19f595daa72544a75a6527f9d02aa032806f91a48076"} Jul 22 13:19:11.124174 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:11.124134 2557 patch_prober.go:28] interesting pod/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 13:19:11.124350 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:11.124206 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 13:19:12.076100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.076069 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:12.124282 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.124243 2557 patch_prober.go:28] interesting pod/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 13:19:12.124609 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.124562 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 13:19:12.225270 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.225233 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca\") pod \"3e4e645c-03f4-4b57-853f-dbf65292429d\" (UID: \"3e4e645c-03f4-4b57-853f-dbf65292429d\") " Jul 22 13:19:12.225586 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.225561 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e4e645c-03f4-4b57-853f-dbf65292429d" (UID: "3e4e645c-03f4-4b57-853f-dbf65292429d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:19:12.326378 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.326339 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4e645c-03f4-4b57-853f-dbf65292429d-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:19:12.934769 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.934739 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" Jul 22 13:19:12.934769 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.934745 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx" event={"ID":"3e4e645c-03f4-4b57-853f-dbf65292429d","Type":"ContainerDied","Data":"fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316"} Jul 22 13:19:12.934990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:12.934788 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb343cdf09676d5b85fe46fe6ce8d0a65e28c8c3ea140e16e308630695afb316" Jul 22 13:19:13.123611 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:13.123568 2557 patch_prober.go:28] interesting pod/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 22 13:19:13.124036 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:13.123621 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 22 13:19:14.128093 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:14.128063 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:19:14.128505 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:14.128335 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:19:14.128975 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:14.128953 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:19:17.872781 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:17.872746 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:17.973804 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:17.973769 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:19.225317 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:19.225267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:19.327469 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:19.327433 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:24.546547 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:24.546480 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:25.039728 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:25.039672 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:28.741272 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:28.741217 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:28.752259 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:28.752221 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx"] Jul 22 13:19:28.755095 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:28.755064 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3d21e-l9wkx"] Jul 22 13:19:29.162075 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:29.161978 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:29.485535 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:29.485506 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4e645c-03f4-4b57-853f-dbf65292429d" path="/var/lib/kubelet/pods/3e4e645c-03f4-4b57-853f-dbf65292429d/volumes" Jul 22 13:19:35.372597 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:35.372563 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:35.476865 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:35.476828 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:35.701521 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:35.701490 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:35.929602 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:35.929542 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:36.049363 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.049272 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9"] Jul 22 13:19:36.049702 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.049683 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4e645c-03f4-4b57-853f-dbf65292429d" containerName="revoke-keys" Jul 22 13:19:36.049822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.049704 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4e645c-03f4-4b57-853f-dbf65292429d" containerName="revoke-keys" Jul 22 13:19:36.049822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.049813 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4e645c-03f4-4b57-853f-dbf65292429d" containerName="revoke-keys" Jul 22 13:19:36.051609 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.051590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:36.054286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.054265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:19:36.054733 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.054704 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:19:36.060496 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.060473 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9"] Jul 22 13:19:36.138344 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.138316 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9\" (UID: \"9c5e0fdb-050e-4de1-8485-efd61d45f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:36.239690 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.239650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9\" (UID: \"9c5e0fdb-050e-4de1-8485-efd61d45f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:36.240350 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.240323 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9\" (UID: \"9c5e0fdb-050e-4de1-8485-efd61d45f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:36.362417 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.362321 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:36.502202 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.502164 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9"] Jul 22 13:19:36.505916 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:19:36.505890 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5e0fdb_050e_4de1_8485_efd61d45f213.slice/crio-76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0 WatchSource:0}: Error finding container 76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0: Status 404 returned error can't find the container with id 76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0 Jul 22 13:19:36.683746 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.683698 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:36.784594 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:36.784562 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:37.048513 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:37.048471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" event={"ID":"9c5e0fdb-050e-4de1-8485-efd61d45f213","Type":"ContainerStarted","Data":"6339fd138486ced5bb367d46659910b29887f8554a6aa1273882ec1de95e2df9"} Jul 22 13:19:37.048704 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:37.048537 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" event={"ID":"9c5e0fdb-050e-4de1-8485-efd61d45f213","Type":"ContainerStarted","Data":"76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0"} Jul 22 13:19:37.063771 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:37.063687 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" podStartSLOduration=1.063669648 podStartE2EDuration="1.063669648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:19:37.063199027 +0000 UTC m=+2316.150688205" watchObservedRunningTime="2026-07-22 13:19:37.063669648 +0000 UTC m=+2316.151158829" Jul 22 13:19:38.054774 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:38.054739 2557 generic.go:356] "Generic (PLEG): container finished" podID="9c5e0fdb-050e-4de1-8485-efd61d45f213" containerID="6339fd138486ced5bb367d46659910b29887f8554a6aa1273882ec1de95e2df9" exitCode=0 Jul 22 13:19:38.055183 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:38.054826 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" event={"ID":"9c5e0fdb-050e-4de1-8485-efd61d45f213","Type":"ContainerDied","Data":"6339fd138486ced5bb367d46659910b29887f8554a6aa1273882ec1de95e2df9"} Jul 22 13:19:39.188599 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:39.188573 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:39.270109 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:39.270069 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca\") pod \"9c5e0fdb-050e-4de1-8485-efd61d45f213\" (UID: \"9c5e0fdb-050e-4de1-8485-efd61d45f213\") " Jul 22 13:19:39.270502 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:39.270476 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c5e0fdb-050e-4de1-8485-efd61d45f213" (UID: "9c5e0fdb-050e-4de1-8485-efd61d45f213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:19:39.370833 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:39.370748 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5e0fdb-050e-4de1-8485-efd61d45f213-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:19:40.066579 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:40.066548 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" Jul 22 13:19:40.066776 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:40.066575 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9" event={"ID":"9c5e0fdb-050e-4de1-8485-efd61d45f213","Type":"ContainerDied","Data":"76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0"} Jul 22 13:19:40.066776 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:40.066627 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cff802acc354fe76a79c1bdc8904dbfadd848e3f9b255478ad33bbeb62daa0" Jul 22 13:19:43.530969 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:43.530937 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:43.596632 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:43.596588 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:51.693993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:51.693947 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:56.139064 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:56.139028 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9"] Jul 22 13:19:56.141528 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:56.141501 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5886b7-dvdm9"] Jul 22 13:19:56.150401 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:56.150352 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:56.591829 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:56.591797 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:19:57.485254 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:19:57.485223 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5e0fdb-050e-4de1-8485-efd61d45f213" path="/var/lib/kubelet/pods/9c5e0fdb-050e-4de1-8485-efd61d45f213/volumes" Jul 22 13:20:02.869414 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:02.869362 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:02.970599 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:02.970564 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:03.128817 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:03.128741 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:20:03.129052 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:03.128996 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" containerID="cri-o://61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4" gracePeriod=30 Jul 22 13:20:03.233639 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:03.233600 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:03.532506 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:03.532459 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:03.773816 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:03.773787 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:04.122292 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:04.122261 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:04.329339 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:04.329307 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:06.952103 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:06.952064 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:07.077000 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:07.076965 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:07.197198 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:07.197163 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:07.318840 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:07.318755 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:07.421325 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:07.421291 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:08.213456 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.213399 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:08.313322 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.313291 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:08.380626 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.380603 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:20:08.441957 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.441913 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.441967 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrt5n\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442009 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442045 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442086 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442107 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442140 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442135 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442444 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442161 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442444 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442224 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs\") pod \"15469df8-a467-4419-a934-5d4907bc16c5\" (UID: \"15469df8-a467-4419-a934-5d4907bc16c5\") " Jul 22 13:20:08.442444 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442324 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:20:08.442588 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442503 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-credential-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.442588 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:20:08.442588 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442576 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:20:08.442743 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.442689 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:20:08.443189 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.443159 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:20:08.444508 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.444482 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:20:08.444624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.444540 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:20:08.444697 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.444674 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:20:08.444766 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.444683 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n" (OuterVolumeSpecName: "kube-api-access-mrt5n") pod "15469df8-a467-4419-a934-5d4907bc16c5" (UID: "15469df8-a467-4419-a934-5d4907bc16c5"). InnerVolumeSpecName "kube-api-access-mrt5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:20:08.543634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543593 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15469df8-a467-4419-a934-5d4907bc16c5-istiod-ca-cert\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543627 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-envoy\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543641 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15469df8-a467-4419-a934-5d4907bc16c5-istio-podinfo\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543653 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-istio-data\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543666 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-socket\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543677 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15469df8-a467-4419-a934-5d4907bc16c5-workload-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543689 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-istio-token\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:08.543918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:08.543701 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrt5n\" (UniqueName: \"kubernetes.io/projected/15469df8-a467-4419-a934-5d4907bc16c5-kube-api-access-mrt5n\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:20:09.201176 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.201144 2557 generic.go:356] "Generic (PLEG): container finished" podID="15469df8-a467-4419-a934-5d4907bc16c5" containerID="61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4" exitCode=0 Jul 22 13:20:09.201356 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.201209 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" Jul 22 13:20:09.201356 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.201226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerDied","Data":"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4"} Jul 22 13:20:09.201356 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.201316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6" event={"ID":"15469df8-a467-4419-a934-5d4907bc16c5","Type":"ContainerDied","Data":"5c54649d283dc851e76328068c482391ec2506546716b5efc5cb26cd85ddfecc"} Jul 22 13:20:09.201356 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.201344 2557 scope.go:117] "RemoveContainer" containerID="61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4" Jul 22 13:20:09.211011 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.210988 2557 scope.go:117] "RemoveContainer" containerID="4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f" Jul 22 13:20:09.221197 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.221169 2557 scope.go:117] "RemoveContainer" containerID="61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4" Jul 22 13:20:09.221584 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:20:09.221487 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4\": container with ID starting with 61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4 not found: ID does not exist" containerID="61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4" Jul 22 13:20:09.221584 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.221514 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4"} err="failed to get container status \"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4\": rpc error: code = NotFound desc = could not find container \"61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4\": container with ID starting with 61a8ee1feda0b034905263b5dc8ddde3cac65297a1f0b466ae5df3885e395db4 not found: ID does not exist" Jul 22 13:20:09.221584 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.221533 2557 scope.go:117] "RemoveContainer" containerID="4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f" Jul 22 13:20:09.221839 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:20:09.221819 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f\": container with ID starting with 4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f not found: ID does not exist" containerID="4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f" Jul 22 13:20:09.221902 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.221844 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f"} err="failed to get container status \"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f\": rpc error: code = NotFound desc = could not find container \"4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f\": container with ID starting with 4403c2eef98c316548059eed0984583b2e0c172426dc6b0a4723b6db9d73f15f not found: ID does not exist" Jul 22 13:20:09.227681 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.227652 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:20:09.232759 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.232724 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5886b7-openshift-default-658469dfbf-rbdc6"] Jul 22 13:20:09.485850 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:09.485764 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15469df8-a467-4419-a934-5d4907bc16c5" path="/var/lib/kubelet/pods/15469df8-a467-4419-a934-5d4907bc16c5/volumes" Jul 22 13:20:11.399156 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:11.399117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:11.777730 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:11.774133 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:12.105061 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:12.104976 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:12.179326 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:12.179295 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:14.890896 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:14.890863 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:14.985771 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:14.985736 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:15.838118 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:15.838081 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:15.972803 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:15.972765 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:17.244582 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:17.244549 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:17.345404 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:17.345368 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:18.077043 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:18.077015 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:18.179531 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:18.179494 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:19.411174 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:19.411132 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:19.523420 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:19.523389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:29.235898 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:29.235866 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:29.336908 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:29.336874 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:30.208685 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:30.208654 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:39.150998 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:39.150963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:39.251898 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:39.251863 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:42.874633 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.874600 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.874917 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.874928 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.874949 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5e0fdb-050e-4de1-8485-efd61d45f213" containerName="revoke-keys" Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.874954 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5e0fdb-050e-4de1-8485-efd61d45f213" containerName="revoke-keys" Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.875006 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:20:42.875017 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.875013 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:20:42.875208 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.875021 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5e0fdb-050e-4de1-8485-efd61d45f213" containerName="revoke-keys" Jul 22 13:20:42.876645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.876629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.879997 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.879968 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"openshift-service-ca.crt\"" Jul 22 13:20:42.879997 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.879986 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"kube-root-ca.crt\"" Jul 22 13:20:42.880190 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.879986 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"default-dockercfg-nctkf\"" Jul 22 13:20:42.880190 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.879968 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"auth-test-model-d24327-kserve-self-signed-certs\"" Jul 22 13:20:42.886641 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.886616 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:20:42.894146 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.894127 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:20:42.945268 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945248 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzwp\" (UniqueName: \"kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.945376 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.945376 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945315 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.945450 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.945450 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:42.945450 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:42.945426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046694 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046672 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzwp\" (UniqueName: \"kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046806 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046727 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046806 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046933 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046988 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.046988 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.046982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.047105 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.047092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.047234 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.047217 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.047357 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.047247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.049067 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.049043 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.049269 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.049253 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.055337 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.055317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzwp\" (UniqueName: \"kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp\") pod \"auth-test-model-d24327-kserve-77f8b55747-q9csn\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.186861 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.186787 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:43.313556 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.313528 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:20:43.315687 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:20:43.315662 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7a36af_75b9_4a33_b8fe_182122a422cf.slice/crio-28c444da3e31071b0f648a945fcabe4f73127ff786e8f871f40a559fb0382751 WatchSource:0}: Error finding container 28c444da3e31071b0f648a945fcabe4f73127ff786e8f871f40a559fb0382751: Status 404 returned error can't find the container with id 28c444da3e31071b0f648a945fcabe4f73127ff786e8f871f40a559fb0382751 Jul 22 13:20:43.359764 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:43.359738 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" event={"ID":"ee7a36af-75b9-4a33-b8fe-182122a422cf","Type":"ContainerStarted","Data":"28c444da3e31071b0f648a945fcabe4f73127ff786e8f871f40a559fb0382751"} Jul 22 13:20:44.368081 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:44.368050 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" event={"ID":"ee7a36af-75b9-4a33-b8fe-182122a422cf","Type":"ContainerStarted","Data":"dca5ede96fcdf97e318edf5d04872233a90606e6e3369559ea35763d48cdae8d"} Jul 22 13:20:44.386024 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:44.385976 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" podStartSLOduration=2.3859603 podStartE2EDuration="2.3859603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:20:44.383704176 +0000 UTC m=+2383.471193368" watchObservedRunningTime="2026-07-22 13:20:44.3859603 +0000 UTC m=+2383.473449481" Jul 22 13:20:53.187477 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:53.187441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:53.187981 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:53.187496 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:53.199552 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:53.199527 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:20:53.420272 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:20:53.420243 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:21:04.364755 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.364695 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:21:04.365139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.365081 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:21:04.365139 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.365095 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15469df8-a467-4419-a934-5d4907bc16c5" containerName="istio-proxy" Jul 22 13:21:04.369360 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.369343 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.373133 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.373109 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"auth-test-model-ffc1de-kserve-self-signed-certs\"" Jul 22 13:21:04.373251 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.373144 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"openshift-service-ca.crt\"" Jul 22 13:21:04.373251 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.373119 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"default-dockercfg-zs8km\"" Jul 22 13:21:04.373251 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.373238 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"kube-root-ca.crt\"" Jul 22 13:21:04.378748 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.378724 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:21:04.391649 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.391629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:21:04.422548 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422497 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4pt\" (UniqueName: \"kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.422548 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422555 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.422788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422594 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.422788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422632 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.422788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.422788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.422693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523467 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4pt\" (UniqueName: \"kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523483 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523510 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523646 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523595 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523858 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.523989 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.523969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.524058 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.524033 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.524116 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.524096 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.525965 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.525942 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.526234 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.526213 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.533066 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.533040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4pt\" (UniqueName: \"kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt\") pod \"auth-test-model-ffc1de-kserve-6858b78587-6fqfr\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.679418 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.679325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:04.812769 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:04.812741 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:21:04.814805 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:21:04.814776 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d650af3_82cc_4e83_b6bc_14a85b5e2802.slice/crio-be3cb5f09a724faa2075358f8b5bc3580ea415b4414b53f3458ad56a1d6fdfd6 WatchSource:0}: Error finding container be3cb5f09a724faa2075358f8b5bc3580ea415b4414b53f3458ad56a1d6fdfd6: Status 404 returned error can't find the container with id be3cb5f09a724faa2075358f8b5bc3580ea415b4414b53f3458ad56a1d6fdfd6 Jul 22 13:21:05.463026 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:05.462993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" event={"ID":"8d650af3-82cc-4e83-b6bc-14a85b5e2802","Type":"ContainerStarted","Data":"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0"} Jul 22 13:21:05.463026 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:05.463033 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" event={"ID":"8d650af3-82cc-4e83-b6bc-14a85b5e2802","Type":"ContainerStarted","Data":"be3cb5f09a724faa2075358f8b5bc3580ea415b4414b53f3458ad56a1d6fdfd6"} Jul 22 13:21:05.483631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:05.483587 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" podStartSLOduration=1.483571585 podStartE2EDuration="1.483571585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:21:05.4811527 +0000 UTC m=+2404.568641885" watchObservedRunningTime="2026-07-22 13:21:05.483571585 +0000 UTC m=+2404.571060767" Jul 22 13:21:14.680190 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:14.680154 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:14.680190 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:14.680208 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:14.692778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:14.692756 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:15.519125 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:15.519095 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:21:43.319986 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:43.319952 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:21:43.423467 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:21:43.423436 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:22:47.814696 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:22:47.814664 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:22:47.876904 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:22:47.876872 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:23:11.911553 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:11.911464 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:23:11.949446 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:11.949416 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:23:12.814615 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:12.814576 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:23:12.846667 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:12.846629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:23:26.136933 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:26.136898 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:23:26.931613 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:26.931567 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" podUID="ee7a36af-75b9-4a33-b8fe-182122a422cf" containerName="main" containerID="cri-o://dca5ede96fcdf97e318edf5d04872233a90606e6e3369559ea35763d48cdae8d" gracePeriod=45 Jul 22 13:23:27.127600 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.127569 2557 generic.go:356] "Generic (PLEG): container finished" podID="ee7a36af-75b9-4a33-b8fe-182122a422cf" containerID="dca5ede96fcdf97e318edf5d04872233a90606e6e3369559ea35763d48cdae8d" exitCode=0 Jul 22 13:23:27.127805 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.127607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" event={"ID":"ee7a36af-75b9-4a33-b8fe-182122a422cf","Type":"ContainerDied","Data":"dca5ede96fcdf97e318edf5d04872233a90606e6e3369559ea35763d48cdae8d"} Jul 22 13:23:27.182940 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.182877 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:23:27.222443 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222413 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222463 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222496 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222546 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222581 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222631 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222613 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zzwp\" (UniqueName: \"kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp\") pod \"ee7a36af-75b9-4a33-b8fe-182122a422cf\" (UID: \"ee7a36af-75b9-4a33-b8fe-182122a422cf\") " Jul 22 13:23:27.222918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222746 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home" (OuterVolumeSpecName: "home") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:27.222918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:27.222918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222863 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:27.222918 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.222876 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.224849 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.224820 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm" (OuterVolumeSpecName: "dshm") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:27.224956 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.224881 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:23:27.225014 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.224983 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp" (OuterVolumeSpecName: "kube-api-access-8zzwp") pod "ee7a36af-75b9-4a33-b8fe-182122a422cf" (UID: "ee7a36af-75b9-4a33-b8fe-182122a422cf"). InnerVolumeSpecName "kube-api-access-8zzwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:23:27.324100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.324042 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7a36af-75b9-4a33-b8fe-182122a422cf-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.324100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.324093 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zzwp\" (UniqueName: \"kubernetes.io/projected/ee7a36af-75b9-4a33-b8fe-182122a422cf-kube-api-access-8zzwp\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.324100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.324104 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.324100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.324113 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.324100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.324123 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7a36af-75b9-4a33-b8fe-182122a422cf-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:27.835103 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:27.834981 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" podUID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" containerName="main" containerID="cri-o://ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0" gracePeriod=45 Jul 22 13:23:28.086517 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.086448 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:23:28.133063 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133034 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133063 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133066 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133334 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133109 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133334 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133135 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133334 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133172 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133334 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133273 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4pt\" (UniqueName: \"kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt\") pod \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\" (UID: \"8d650af3-82cc-4e83-b6bc-14a85b5e2802\") " Jul 22 13:23:28.133549 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133388 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:28.133612 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133585 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.133801 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133769 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:28.133801 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.133781 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home" (OuterVolumeSpecName: "home") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:28.134445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.134421 2557 generic.go:356] "Generic (PLEG): container finished" podID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" containerID="ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0" exitCode=0 Jul 22 13:23:28.134555 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.134522 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" event={"ID":"8d650af3-82cc-4e83-b6bc-14a85b5e2802","Type":"ContainerDied","Data":"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0"} Jul 22 13:23:28.134555 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.134534 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" Jul 22 13:23:28.135171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.134562 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr" event={"ID":"8d650af3-82cc-4e83-b6bc-14a85b5e2802","Type":"ContainerDied","Data":"be3cb5f09a724faa2075358f8b5bc3580ea415b4414b53f3458ad56a1d6fdfd6"} Jul 22 13:23:28.135171 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.134585 2557 scope.go:117] "RemoveContainer" containerID="ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0" Jul 22 13:23:28.136100 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.136076 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:23:28.136197 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.136102 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" event={"ID":"ee7a36af-75b9-4a33-b8fe-182122a422cf","Type":"ContainerDied","Data":"28c444da3e31071b0f648a945fcabe4f73127ff786e8f871f40a559fb0382751"} Jul 22 13:23:28.136197 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.136139 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn" Jul 22 13:23:28.136197 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.136178 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt" (OuterVolumeSpecName: "kube-api-access-st4pt") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "kube-api-access-st4pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:23:28.136366 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.136255 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm" (OuterVolumeSpecName: "dshm") pod "8d650af3-82cc-4e83-b6bc-14a85b5e2802" (UID: "8d650af3-82cc-4e83-b6bc-14a85b5e2802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:23:28.154254 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.154235 2557 scope.go:117] "RemoveContainer" containerID="ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0" Jul 22 13:23:28.154574 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:23:28.154552 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0\": container with ID starting with ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0 not found: ID does not exist" containerID="ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0" Jul 22 13:23:28.154666 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.154587 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0"} err="failed to get container status \"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0\": rpc error: code = NotFound desc = could not find container \"ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0\": container with ID starting with ce45b854970523a8028ca5355014dec7832bce8e8c8f958cc90bb17b9802b2b0 not found: ID does not exist" Jul 22 13:23:28.154666 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.154614 2557 scope.go:117] "RemoveContainer" containerID="dca5ede96fcdf97e318edf5d04872233a90606e6e3369559ea35763d48cdae8d" Jul 22 13:23:28.167566 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.167536 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:23:28.172526 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.172499 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/auth-test-model-d24327-kserve-77f8b55747-q9csn"] Jul 22 13:23:28.234082 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.234042 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.234082 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.234075 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d650af3-82cc-4e83-b6bc-14a85b5e2802-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.234082 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.234086 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.234538 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.234101 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4pt\" (UniqueName: \"kubernetes.io/projected/8d650af3-82cc-4e83-b6bc-14a85b5e2802-kube-api-access-st4pt\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.234538 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.234113 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d650af3-82cc-4e83-b6bc-14a85b5e2802-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:23:28.458292 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.458257 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:23:28.462262 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:28.462222 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/auth-test-model-ffc1de-kserve-6858b78587-6fqfr"] Jul 22 13:23:29.485364 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:29.485330 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" path="/var/lib/kubelet/pods/8d650af3-82cc-4e83-b6bc-14a85b5e2802/volumes" Jul 22 13:23:29.485763 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:29.485735 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee7a36af-75b9-4a33-b8fe-182122a422cf" path="/var/lib/kubelet/pods/ee7a36af-75b9-4a33-b8fe-182122a422cf/volumes" Jul 22 13:23:58.891020 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.890983 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891305 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" containerName="main" Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891315 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" containerName="main" Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891331 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7a36af-75b9-4a33-b8fe-182122a422cf" containerName="main" Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891337 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7a36af-75b9-4a33-b8fe-182122a422cf" containerName="main" Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891398 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d650af3-82cc-4e83-b6bc-14a85b5e2802" containerName="main" Jul 22 13:23:58.891498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.891408 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee7a36af-75b9-4a33-b8fe-182122a422cf" containerName="main" Jul 22 13:23:58.894500 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.894484 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:58.898870 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.898843 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"sub-test-model-ffc1de-kserve-self-signed-certs\"" Jul 22 13:23:58.898870 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.898857 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"default-dockercfg-zs8km\"" Jul 22 13:23:58.899078 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.898846 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"kube-root-ca.crt\"" Jul 22 13:23:58.899078 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.898909 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"openshift-service-ca.crt\"" Jul 22 13:23:58.904889 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.904865 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:23:58.925283 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:58.925251 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:23:59.008645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdg65\" (UniqueName: \"kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.008645 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008649 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.008940 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.008940 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.008940 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008865 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.009084 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.008948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.109903 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.109870 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.109903 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.109907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.109953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.109985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.110011 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdg65\" (UniqueName: \"kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.110035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.110363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110449 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.110428 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.110494 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.110460 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.112230 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.112208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.112394 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.112379 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.118209 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.118188 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdg65\" (UniqueName: \"kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65\") pod \"sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.203578 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.203552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:23:59.340661 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:23:59.340631 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:23:59.341487 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:23:59.341466 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6431d94f_87e6_4da3_a210_59d76038b1e2.slice/crio-5861dafd496e6582f997d38d5bf6c7506909af1db15a88c2bc5c8bd878f5e1d8 WatchSource:0}: Error finding container 5861dafd496e6582f997d38d5bf6c7506909af1db15a88c2bc5c8bd878f5e1d8: Status 404 returned error can't find the container with id 5861dafd496e6582f997d38d5bf6c7506909af1db15a88c2bc5c8bd878f5e1d8 Jul 22 13:24:00.291728 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:00.291674 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" event={"ID":"6431d94f-87e6-4da3-a210-59d76038b1e2","Type":"ContainerStarted","Data":"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4"} Jul 22 13:24:00.292141 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:00.291736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" event={"ID":"6431d94f-87e6-4da3-a210-59d76038b1e2","Type":"ContainerStarted","Data":"5861dafd496e6582f997d38d5bf6c7506909af1db15a88c2bc5c8bd878f5e1d8"} Jul 22 13:24:00.311307 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:00.311258 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" podStartSLOduration=2.311240884 podStartE2EDuration="2.311240884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:24:00.31025051 +0000 UTC m=+2579.397739685" watchObservedRunningTime="2026-07-22 13:24:00.311240884 +0000 UTC m=+2579.398730065" Jul 22 13:24:01.716514 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:01.716482 2557 scope.go:117] "RemoveContainer" containerID="72defdbe47aaffdfea3a7ede4d7c60026839d4ea5794ad23158f0754d20c52b6" Jul 22 13:24:09.203825 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:09.203783 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:09.203825 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:09.203831 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:09.217268 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:09.217236 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:09.344191 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:09.344162 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:38.392480 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:38.392443 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:24:39.180757 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:39.180719 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:24:39.194367 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:39.194308 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:24:50.671783 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.671743 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:24:50.674667 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.674647 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.677532 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.677499 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"default-dockercfg-nctkf\"" Jul 22 13:24:50.678512 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.678497 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"kube-root-ca.crt\"" Jul 22 13:24:50.678765 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.678747 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"rate-test-model-d24327-kserve-self-signed-certs\"" Jul 22 13:24:50.678860 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.678786 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"openshift-service-ca.crt\"" Jul 22 13:24:50.688953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.688917 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:24:50.705071 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.705037 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:24:50.774155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774089 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.774155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774157 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.774437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774233 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8r4\" (UniqueName: \"kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.774437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774287 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.774437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.774437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.774435 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875784 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875701 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875877 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875895 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.875993 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.875918 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8r4\" (UniqueName: \"kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.876294 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.876269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.876352 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.876281 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.876352 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.876335 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.878059 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.878041 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.878402 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.878384 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.885164 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.885139 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8r4\" (UniqueName: \"kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4\") pod \"rate-test-model-d24327-kserve-57547677dc-vbk4v\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:50.985513 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:50.985476 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:24:51.117417 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:51.117390 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:24:51.119502 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:24:51.119461 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb534030d_b318_43ed_8599_e0fbf719a7ca.slice/crio-f6826583e440dea9f4a16b4ea7993fda45468f9e797de1c8e67585b6b04d42cd WatchSource:0}: Error finding container f6826583e440dea9f4a16b4ea7993fda45468f9e797de1c8e67585b6b04d42cd: Status 404 returned error can't find the container with id f6826583e440dea9f4a16b4ea7993fda45468f9e797de1c8e67585b6b04d42cd Jul 22 13:24:51.527946 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:51.527915 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" event={"ID":"b534030d-b318-43ed-8599-e0fbf719a7ca","Type":"ContainerStarted","Data":"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50"} Jul 22 13:24:51.527946 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:51.527959 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" event={"ID":"b534030d-b318-43ed-8599-e0fbf719a7ca","Type":"ContainerStarted","Data":"f6826583e440dea9f4a16b4ea7993fda45468f9e797de1c8e67585b6b04d42cd"} Jul 22 13:24:51.556453 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:51.556402 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" podStartSLOduration=1.556377796 podStartE2EDuration="1.556377796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:24:51.555351449 +0000 UTC m=+2630.642840642" watchObservedRunningTime="2026-07-22 13:24:51.556377796 +0000 UTC m=+2630.643866976" Jul 22 13:24:54.200091 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.200045 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" podUID="6431d94f-87e6-4da3-a210-59d76038b1e2" containerName="main" containerID="cri-o://9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4" gracePeriod=45 Jul 22 13:24:54.447178 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.447152 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:54.507404 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507309 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507404 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507353 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507404 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507394 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdg65\" (UniqueName: \"kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507420 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507469 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507494 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm\") pod \"6431d94f-87e6-4da3-a210-59d76038b1e2\" (UID: \"6431d94f-87e6-4da3-a210-59d76038b1e2\") " Jul 22 13:24:54.507701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507651 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home" (OuterVolumeSpecName: "home") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:54.507701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507660 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:54.507953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507777 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:54.507953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507792 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.507953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.507809 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.509808 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.509771 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm" (OuterVolumeSpecName: "dshm") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:54.509930 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.509815 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65" (OuterVolumeSpecName: "kube-api-access-vdg65") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "kube-api-access-vdg65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:24:54.509930 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.509840 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6431d94f-87e6-4da3-a210-59d76038b1e2" (UID: "6431d94f-87e6-4da3-a210-59d76038b1e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:24:54.549778 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.549740 2557 generic.go:356] "Generic (PLEG): container finished" podID="6431d94f-87e6-4da3-a210-59d76038b1e2" containerID="9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4" exitCode=0 Jul 22 13:24:54.549943 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.549824 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" event={"ID":"6431d94f-87e6-4da3-a210-59d76038b1e2","Type":"ContainerDied","Data":"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4"} Jul 22 13:24:54.549943 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.549853 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" event={"ID":"6431d94f-87e6-4da3-a210-59d76038b1e2","Type":"ContainerDied","Data":"5861dafd496e6582f997d38d5bf6c7506909af1db15a88c2bc5c8bd878f5e1d8"} Jul 22 13:24:54.549943 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.549869 2557 scope.go:117] "RemoveContainer" containerID="9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4" Jul 22 13:24:54.549943 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.549871 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp" Jul 22 13:24:54.558587 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.558567 2557 scope.go:117] "RemoveContainer" containerID="9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4" Jul 22 13:24:54.558880 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:24:54.558861 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4\": container with ID starting with 9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4 not found: ID does not exist" containerID="9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4" Jul 22 13:24:54.558944 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.558894 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4"} err="failed to get container status \"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4\": rpc error: code = NotFound desc = could not find container \"9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4\": container with ID starting with 9700f2d8111ccb3d0c7cf043ce69c5b582fa84ed9b9b5a24de2aa824bb5248a4 not found: ID does not exist" Jul 22 13:24:54.572156 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.572131 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:24:54.574311 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.574290 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/sub-test-model-ffc1de-kserve-cf4fcbd7c-snphp"] Jul 22 13:24:54.608922 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.608894 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdg65\" (UniqueName: \"kubernetes.io/projected/6431d94f-87e6-4da3-a210-59d76038b1e2-kube-api-access-vdg65\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.608922 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.608918 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431d94f-87e6-4da3-a210-59d76038b1e2-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.609108 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.608932 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.609108 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:54.608943 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431d94f-87e6-4da3-a210-59d76038b1e2-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:24:55.486184 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:24:55.486150 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6431d94f-87e6-4da3-a210-59d76038b1e2" path="/var/lib/kubelet/pods/6431d94f-87e6-4da3-a210-59d76038b1e2/volumes" Jul 22 13:25:00.985810 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:00.985779 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:25:00.986308 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:00.985828 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:25:00.998272 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:00.998249 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:25:01.593944 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:01.593914 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:25:01.741592 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:01.741562 2557 scope.go:117] "RemoveContainer" containerID="3996b0be9d77d2c65c5253c3bf406f5fc187c8874ae1357640c5dc992bf3f564" Jul 22 13:25:15.624293 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:15.624256 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:18.102381 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:18.102346 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:18.203990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:18.203956 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:30.023164 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:30.023123 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:30.131249 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:30.131219 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:39.899251 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:39.899217 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:40.006449 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:40.006418 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:53.986096 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:53.986008 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:25:54.011822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:54.011785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:55.013213 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:55.013179 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:25:56.029843 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.029805 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089127358128"] Jul 22 13:25:56.030252 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.030131 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6431d94f-87e6-4da3-a210-59d76038b1e2" containerName="main" Jul 22 13:25:56.030252 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.030143 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6431d94f-87e6-4da3-a210-59d76038b1e2" containerName="main" Jul 22 13:25:56.030252 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.030209 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6431d94f-87e6-4da3-a210-59d76038b1e2" containerName="main" Jul 22 13:25:56.032889 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.032870 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:56.035999 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.035984 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bqc8f\"" Jul 22 13:25:56.039469 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.039436 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127358128"] Jul 22 13:25:56.154249 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.154201 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2j9\" (UniqueName: \"kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9\") pod \"test-curl-7777-140089127358128\" (UID: \"2bc33d6b-c80c-41de-8723-155f348f0e70\") " pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:56.254909 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.254866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2j9\" (UniqueName: \"kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9\") pod \"test-curl-7777-140089127358128\" (UID: \"2bc33d6b-c80c-41de-8723-155f348f0e70\") " pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:56.265683 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.265657 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2j9\" (UniqueName: \"kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9\") pod \"test-curl-7777-140089127358128\" (UID: \"2bc33d6b-c80c-41de-8723-155f348f0e70\") " pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:56.342742 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.342626 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:56.489777 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.489749 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127358128"] Jul 22 13:25:56.492208 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:25:56.492180 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bc33d6b_c80c_41de_8723_155f348f0e70.slice/crio-f203efbcde2cbef78fc081d2a9a549c2bf86e4047a03dd9de2d88e71fe924528 WatchSource:0}: Error finding container f203efbcde2cbef78fc081d2a9a549c2bf86e4047a03dd9de2d88e71fe924528: Status 404 returned error can't find the container with id f203efbcde2cbef78fc081d2a9a549c2bf86e4047a03dd9de2d88e71fe924528 Jul 22 13:25:56.493486 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.493464 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:25:56.824960 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:56.824928 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127358128" event={"ID":"2bc33d6b-c80c-41de-8723-155f348f0e70","Type":"ContainerStarted","Data":"f203efbcde2cbef78fc081d2a9a549c2bf86e4047a03dd9de2d88e71fe924528"} Jul 22 13:25:57.831150 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:57.831116 2557 generic.go:356] "Generic (PLEG): container finished" podID="2bc33d6b-c80c-41de-8723-155f348f0e70" containerID="e5903a1696aa66c16bdc0dd8f2cc3f195ce154f095a44c8d8f8192b4f79c0d9c" exitCode=0 Jul 22 13:25:57.831536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:57.831177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127358128" event={"ID":"2bc33d6b-c80c-41de-8723-155f348f0e70","Type":"ContainerDied","Data":"e5903a1696aa66c16bdc0dd8f2cc3f195ce154f095a44c8d8f8192b4f79c0d9c"} Jul 22 13:25:58.983772 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:58.983740 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:25:59.007608 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.007552 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089127358128_2bc33d6b-c80c-41de-8723-155f348f0e70/test-curl-7777-140089127358128/0.log" Jul 22 13:25:59.033934 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.033887 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089127358128"] Jul 22 13:25:59.038077 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.038049 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089127358128"] Jul 22 13:25:59.081379 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.081348 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2j9\" (UniqueName: \"kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9\") pod \"2bc33d6b-c80c-41de-8723-155f348f0e70\" (UID: \"2bc33d6b-c80c-41de-8723-155f348f0e70\") " Jul 22 13:25:59.083510 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.083483 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9" (OuterVolumeSpecName: "kube-api-access-zn2j9") pod "2bc33d6b-c80c-41de-8723-155f348f0e70" (UID: "2bc33d6b-c80c-41de-8723-155f348f0e70"). InnerVolumeSpecName "kube-api-access-zn2j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:25:59.170889 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.170849 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089127357104"] Jul 22 13:25:59.171255 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.171235 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc33d6b-c80c-41de-8723-155f348f0e70" containerName="test-curl-7777-140089127358128" Jul 22 13:25:59.171343 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.171258 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc33d6b-c80c-41de-8723-155f348f0e70" containerName="test-curl-7777-140089127358128" Jul 22 13:25:59.171398 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.171357 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc33d6b-c80c-41de-8723-155f348f0e70" containerName="test-curl-7777-140089127358128" Jul 22 13:25:59.174227 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.174207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:25:59.182670 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.182636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45zg\" (UniqueName: \"kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg\") pod \"test-curl-7777-140089127357104\" (UID: \"29c5e536-e6b5-48bb-898d-41300c5c02ea\") " pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:25:59.182849 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.182829 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2j9\" (UniqueName: \"kubernetes.io/projected/2bc33d6b-c80c-41de-8723-155f348f0e70-kube-api-access-zn2j9\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:25:59.184464 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.184441 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127357104"] Jul 22 13:25:59.283228 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.283183 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45zg\" (UniqueName: \"kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg\") pod \"test-curl-7777-140089127357104\" (UID: \"29c5e536-e6b5-48bb-898d-41300c5c02ea\") " pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:25:59.293224 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.293192 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45zg\" (UniqueName: \"kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg\") pod \"test-curl-7777-140089127357104\" (UID: \"29c5e536-e6b5-48bb-898d-41300c5c02ea\") " pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:25:59.482939 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.482914 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:25:59.489890 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.487644 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc33d6b-c80c-41de-8723-155f348f0e70" path="/var/lib/kubelet/pods/2bc33d6b-c80c-41de-8723-155f348f0e70/volumes" Jul 22 13:25:59.614297 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.614190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127357104"] Jul 22 13:25:59.616789 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:25:59.616765 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c5e536_e6b5_48bb_898d_41300c5c02ea.slice/crio-18b51fe20a279e4151fe6ede4c7272f3636a54fe3dd56a6f21023b281fb8fc84 WatchSource:0}: Error finding container 18b51fe20a279e4151fe6ede4c7272f3636a54fe3dd56a6f21023b281fb8fc84: Status 404 returned error can't find the container with id 18b51fe20a279e4151fe6ede4c7272f3636a54fe3dd56a6f21023b281fb8fc84 Jul 22 13:25:59.840634 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.840553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127357104" event={"ID":"29c5e536-e6b5-48bb-898d-41300c5c02ea","Type":"ContainerStarted","Data":"18b51fe20a279e4151fe6ede4c7272f3636a54fe3dd56a6f21023b281fb8fc84"} Jul 22 13:25:59.841782 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.841758 2557 scope.go:117] "RemoveContainer" containerID="e5903a1696aa66c16bdc0dd8f2cc3f195ce154f095a44c8d8f8192b4f79c0d9c" Jul 22 13:25:59.841886 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:25:59.841765 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127358128" Jul 22 13:26:00.848575 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:00.848537 2557 generic.go:356] "Generic (PLEG): container finished" podID="29c5e536-e6b5-48bb-898d-41300c5c02ea" containerID="e817d9c7d1e7787ab2a1bb9ec5e8c61bba6e300e786b90fb504c15e406fefba9" exitCode=0 Jul 22 13:26:00.848991 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:00.848612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127357104" event={"ID":"29c5e536-e6b5-48bb-898d-41300c5c02ea","Type":"ContainerDied","Data":"e817d9c7d1e7787ab2a1bb9ec5e8c61bba6e300e786b90fb504c15e406fefba9"} Jul 22 13:26:01.762388 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:01.762353 2557 scope.go:117] "RemoveContainer" containerID="6339fd138486ced5bb367d46659910b29887f8554a6aa1273882ec1de95e2df9" Jul 22 13:26:01.770455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:01.770435 2557 scope.go:117] "RemoveContainer" containerID="2e11f1f1222502ee61be19f595daa72544a75a6527f9d02aa032806f91a48076" Jul 22 13:26:01.985621 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:01.985592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:26:02.003018 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.002990 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t45zg\" (UniqueName: \"kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg\") pod \"29c5e536-e6b5-48bb-898d-41300c5c02ea\" (UID: \"29c5e536-e6b5-48bb-898d-41300c5c02ea\") " Jul 22 13:26:02.004980 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.004960 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg" (OuterVolumeSpecName: "kube-api-access-t45zg") pod "29c5e536-e6b5-48bb-898d-41300c5c02ea" (UID: "29c5e536-e6b5-48bb-898d-41300c5c02ea"). InnerVolumeSpecName "kube-api-access-t45zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:02.022728 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.022645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089127357104_29c5e536-e6b5-48bb-898d-41300c5c02ea/test-curl-7777-140089127357104/0.log" Jul 22 13:26:02.059981 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.059950 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089127357104"] Jul 22 13:26:02.062637 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.062612 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089127357104"] Jul 22 13:26:02.103573 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.103546 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45zg\" (UniqueName: \"kubernetes.io/projected/29c5e536-e6b5-48bb-898d-41300c5c02ea-kube-api-access-t45zg\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:02.241561 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.241526 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:02.241926 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.241912 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c5e536-e6b5-48bb-898d-41300c5c02ea" containerName="test-curl-7777-140089127357104" Jul 22 13:26:02.241977 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.241929 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c5e536-e6b5-48bb-898d-41300c5c02ea" containerName="test-curl-7777-140089127357104" Jul 22 13:26:02.242013 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.241990 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c5e536-e6b5-48bb-898d-41300c5c02ea" containerName="test-curl-7777-140089127357104" Jul 22 13:26:02.244906 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.244888 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:02.253901 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.253878 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:02.305245 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.305154 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf8k\" (UniqueName: \"kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k\") pod \"test-curl-7777-140089127466736\" (UID: \"83f0092d-4a91-4c19-916d-0791dbf1d9ed\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:02.405624 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.405592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf8k\" (UniqueName: \"kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k\") pod \"test-curl-7777-140089127466736\" (UID: \"83f0092d-4a91-4c19-916d-0791dbf1d9ed\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:02.415009 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.414986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf8k\" (UniqueName: \"kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k\") pod \"test-curl-7777-140089127466736\" (UID: \"83f0092d-4a91-4c19-916d-0791dbf1d9ed\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:02.555040 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.555005 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:02.681794 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.681767 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:02.683058 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:02.683035 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83f0092d_4a91_4c19_916d_0791dbf1d9ed.slice/crio-6805fa3dccabe7386f2325afef30910f10bfdb1985bf986d2e43227549ed95ca WatchSource:0}: Error finding container 6805fa3dccabe7386f2325afef30910f10bfdb1985bf986d2e43227549ed95ca: Status 404 returned error can't find the container with id 6805fa3dccabe7386f2325afef30910f10bfdb1985bf986d2e43227549ed95ca Jul 22 13:26:02.858557 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.858477 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:26:02.858557 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.858508 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b51fe20a279e4151fe6ede4c7272f3636a54fe3dd56a6f21023b281fb8fc84" Jul 22 13:26:02.859691 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:02.859666 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127466736" event={"ID":"83f0092d-4a91-4c19-916d-0791dbf1d9ed","Type":"ContainerStarted","Data":"6805fa3dccabe7386f2325afef30910f10bfdb1985bf986d2e43227549ed95ca"} Jul 22 13:26:02.869203 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:26:02.869179 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-140089127357104\" is forbidden: User \"system:node:ip-10-0-137-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-115.ec2.internal' and this object" podUID="29c5e536-e6b5-48bb-898d-41300c5c02ea" pod="opendatahub/test-curl-7777-140089127357104" Jul 22 13:26:03.485546 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:03.485508 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c5e536-e6b5-48bb-898d-41300c5c02ea" path="/var/lib/kubelet/pods/29c5e536-e6b5-48bb-898d-41300c5c02ea/volumes" Jul 22 13:26:03.865836 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:03.865737 2557 generic.go:356] "Generic (PLEG): container finished" podID="83f0092d-4a91-4c19-916d-0791dbf1d9ed" containerID="9bcb7bd90576f2d1eb341ee39335c5b9a1e6bf7e0fd11149c33ccc00c47be65f" exitCode=0 Jul 22 13:26:03.865836 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:03.865817 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127466736" event={"ID":"83f0092d-4a91-4c19-916d-0791dbf1d9ed","Type":"ContainerDied","Data":"9bcb7bd90576f2d1eb341ee39335c5b9a1e6bf7e0fd11149c33ccc00c47be65f"} Jul 22 13:26:05.001924 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.001895 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.026888 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.026856 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089127466736_83f0092d-4a91-4c19-916d-0791dbf1d9ed/test-curl-7777-140089127466736/0.log" Jul 22 13:26:05.028990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.028968 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jf8k\" (UniqueName: \"kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k\") pod \"83f0092d-4a91-4c19-916d-0791dbf1d9ed\" (UID: \"83f0092d-4a91-4c19-916d-0791dbf1d9ed\") " Jul 22 13:26:05.031076 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.031050 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k" (OuterVolumeSpecName: "kube-api-access-8jf8k") pod "83f0092d-4a91-4c19-916d-0791dbf1d9ed" (UID: "83f0092d-4a91-4c19-916d-0791dbf1d9ed"). InnerVolumeSpecName "kube-api-access-8jf8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:05.056874 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.056847 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:05.060718 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.060687 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:05.130603 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.130528 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jf8k\" (UniqueName: \"kubernetes.io/projected/83f0092d-4a91-4c19-916d-0791dbf1d9ed-kube-api-access-8jf8k\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:05.181562 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.181531 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:05.181907 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.181893 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f0092d-4a91-4c19-916d-0791dbf1d9ed" containerName="test-curl-7777-140089127466736" Jul 22 13:26:05.181955 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.181909 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f0092d-4a91-4c19-916d-0791dbf1d9ed" containerName="test-curl-7777-140089127466736" Jul 22 13:26:05.181996 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.181970 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f0092d-4a91-4c19-916d-0791dbf1d9ed" containerName="test-curl-7777-140089127466736" Jul 22 13:26:05.184840 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.184823 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.190552 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.190523 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:05.231765 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.231706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckrh\" (UniqueName: \"kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh\") pod \"test-curl-7777-140089127466736\" (UID: \"8ffba4c5-6f69-45d1-93a4-b59c0be9a039\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.332368 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.332327 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckrh\" (UniqueName: \"kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh\") pod \"test-curl-7777-140089127466736\" (UID: \"8ffba4c5-6f69-45d1-93a4-b59c0be9a039\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.341298 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.341267 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckrh\" (UniqueName: \"kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh\") pod \"test-curl-7777-140089127466736\" (UID: \"8ffba4c5-6f69-45d1-93a4-b59c0be9a039\") " pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.486135 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.486102 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f0092d-4a91-4c19-916d-0791dbf1d9ed" path="/var/lib/kubelet/pods/83f0092d-4a91-4c19-916d-0791dbf1d9ed/volumes" Jul 22 13:26:05.495342 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.495320 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.624155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.624129 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:05.626255 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:05.626223 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ffba4c5_6f69_45d1_93a4_b59c0be9a039.slice/crio-bf40e9f4df23e2eabc5be412d1dc283b52c87506c78bf0f0111c1c92f08f1a9a WatchSource:0}: Error finding container bf40e9f4df23e2eabc5be412d1dc283b52c87506c78bf0f0111c1c92f08f1a9a: Status 404 returned error can't find the container with id bf40e9f4df23e2eabc5be412d1dc283b52c87506c78bf0f0111c1c92f08f1a9a Jul 22 13:26:05.877143 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.877063 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:05.877143 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.877099 2557 scope.go:117] "RemoveContainer" containerID="9bcb7bd90576f2d1eb341ee39335c5b9a1e6bf7e0fd11149c33ccc00c47be65f" Jul 22 13:26:05.878609 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:05.878582 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127466736" event={"ID":"8ffba4c5-6f69-45d1-93a4-b59c0be9a039","Type":"ContainerStarted","Data":"bf40e9f4df23e2eabc5be412d1dc283b52c87506c78bf0f0111c1c92f08f1a9a"} Jul 22 13:26:06.884702 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:06.884664 2557 generic.go:356] "Generic (PLEG): container finished" podID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" containerID="3d80c86f9d9cd2b3d24f76d5a3525737567328aa508cde6236d5f9ec7fa6847d" exitCode=0 Jul 22 13:26:06.885085 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:06.884748 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089127466736" event={"ID":"8ffba4c5-6f69-45d1-93a4-b59c0be9a039","Type":"ContainerDied","Data":"3d80c86f9d9cd2b3d24f76d5a3525737567328aa508cde6236d5f9ec7fa6847d"} Jul 22 13:26:08.025859 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.025830 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:08.050463 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.050435 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089127466736_8ffba4c5-6f69-45d1-93a4-b59c0be9a039/test-curl-7777-140089127466736/0.log" Jul 22 13:26:08.058622 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.058598 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mckrh\" (UniqueName: \"kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh\") pod \"8ffba4c5-6f69-45d1-93a4-b59c0be9a039\" (UID: \"8ffba4c5-6f69-45d1-93a4-b59c0be9a039\") " Jul 22 13:26:08.060697 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.060676 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh" (OuterVolumeSpecName: "kube-api-access-mckrh") pod "8ffba4c5-6f69-45d1-93a4-b59c0be9a039" (UID: "8ffba4c5-6f69-45d1-93a4-b59c0be9a039"). InnerVolumeSpecName "kube-api-access-mckrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:08.078263 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.078222 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:08.081955 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.081930 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089127466736"] Jul 22 13:26:08.159963 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.159886 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mckrh\" (UniqueName: \"kubernetes.io/projected/8ffba4c5-6f69-45d1-93a4-b59c0be9a039-kube-api-access-mckrh\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:08.213206 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.213175 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:08.213566 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.213551 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" containerName="test-curl-7777-140089127466736" Jul 22 13:26:08.213611 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.213568 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" containerName="test-curl-7777-140089127466736" Jul 22 13:26:08.213644 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.213630 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" containerName="test-curl-7777-140089127466736" Jul 22 13:26:08.216492 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.216476 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:08.224894 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.224866 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:08.261352 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.261313 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnzz\" (UniqueName: \"kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz\") pod \"test-curl-7777-140089130910608\" (UID: \"2e0df439-5f29-44ee-b26b-d8623bdb412d\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:08.362616 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.362576 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnzz\" (UniqueName: \"kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz\") pod \"test-curl-7777-140089130910608\" (UID: \"2e0df439-5f29-44ee-b26b-d8623bdb412d\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:08.371996 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.371969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnzz\" (UniqueName: \"kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz\") pod \"test-curl-7777-140089130910608\" (UID: \"2e0df439-5f29-44ee-b26b-d8623bdb412d\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:08.526422 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.526370 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:08.663525 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.663500 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:08.665974 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:08.665949 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e0df439_5f29_44ee_b26b_d8623bdb412d.slice/crio-794de08ea3b890dffb0cdec60655e76280d15b35671ae2630729be3b843a9573 WatchSource:0}: Error finding container 794de08ea3b890dffb0cdec60655e76280d15b35671ae2630729be3b843a9573: Status 404 returned error can't find the container with id 794de08ea3b890dffb0cdec60655e76280d15b35671ae2630729be3b843a9573 Jul 22 13:26:08.895289 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.895204 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:08.895289 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.895251 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf40e9f4df23e2eabc5be412d1dc283b52c87506c78bf0f0111c1c92f08f1a9a" Jul 22 13:26:08.896370 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:08.896342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089130910608" event={"ID":"2e0df439-5f29-44ee-b26b-d8623bdb412d","Type":"ContainerStarted","Data":"794de08ea3b890dffb0cdec60655e76280d15b35671ae2630729be3b843a9573"} Jul 22 13:26:08.906369 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:26:08.906336 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-140089127466736\" is forbidden: User \"system:node:ip-10-0-137-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-115.ec2.internal' and this object" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:09.006788 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.006754 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" podUID="b534030d-b318-43ed-8599-e0fbf719a7ca" containerName="main" containerID="cri-o://5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50" gracePeriod=45 Jul 22 13:26:09.278386 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.278354 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:26:09.301185 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:26:09.301119 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-140089127466736\" is forbidden: User \"system:node:ip-10-0-137-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-115.ec2.internal' and this object" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" pod="opendatahub/test-curl-7777-140089127466736" Jul 22 13:26:09.371136 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371100 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx8r4\" (UniqueName: \"kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371156 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371181 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371239 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371419 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371293 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371419 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371348 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir\") pod \"b534030d-b318-43ed-8599-e0fbf719a7ca\" (UID: \"b534030d-b318-43ed-8599-e0fbf719a7ca\") " Jul 22 13:26:09.371541 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371510 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home" (OuterVolumeSpecName: "home") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:09.371774 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371748 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:09.371865 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.371784 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:09.373403 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.373380 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4" (OuterVolumeSpecName: "kube-api-access-dx8r4") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "kube-api-access-dx8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:09.373403 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.373398 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:26:09.373607 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.373580 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm" (OuterVolumeSpecName: "dshm") pod "b534030d-b318-43ed-8599-e0fbf719a7ca" (UID: "b534030d-b318-43ed-8599-e0fbf719a7ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:09.472828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472786 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8r4\" (UniqueName: \"kubernetes.io/projected/b534030d-b318-43ed-8599-e0fbf719a7ca-kube-api-access-dx8r4\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.472828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472818 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-model-cache\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.472828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472828 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-dshm\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.472828 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472836 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-home\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.473073 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472845 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b534030d-b318-43ed-8599-e0fbf719a7ca-tls-certs\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.473073 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.472852 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b534030d-b318-43ed-8599-e0fbf719a7ca-tmp-dir\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:09.486510 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.486476 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffba4c5-6f69-45d1-93a4-b59c0be9a039" path="/var/lib/kubelet/pods/8ffba4c5-6f69-45d1-93a4-b59c0be9a039/volumes" Jul 22 13:26:09.903050 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.902968 2557 generic.go:356] "Generic (PLEG): container finished" podID="2e0df439-5f29-44ee-b26b-d8623bdb412d" containerID="551ebfde4327b814dd1769bd1f7c8e6a31be565d79cccb35117ec71f80e19d0d" exitCode=0 Jul 22 13:26:09.903215 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.903052 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089130910608" event={"ID":"2e0df439-5f29-44ee-b26b-d8623bdb412d","Type":"ContainerDied","Data":"551ebfde4327b814dd1769bd1f7c8e6a31be565d79cccb35117ec71f80e19d0d"} Jul 22 13:26:09.904329 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.904302 2557 generic.go:356] "Generic (PLEG): container finished" podID="b534030d-b318-43ed-8599-e0fbf719a7ca" containerID="5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50" exitCode=0 Jul 22 13:26:09.904445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.904342 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" Jul 22 13:26:09.904445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.904376 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" event={"ID":"b534030d-b318-43ed-8599-e0fbf719a7ca","Type":"ContainerDied","Data":"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50"} Jul 22 13:26:09.904445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.904421 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v" event={"ID":"b534030d-b318-43ed-8599-e0fbf719a7ca","Type":"ContainerDied","Data":"f6826583e440dea9f4a16b4ea7993fda45468f9e797de1c8e67585b6b04d42cd"} Jul 22 13:26:09.904445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.904441 2557 scope.go:117] "RemoveContainer" containerID="5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50" Jul 22 13:26:09.912846 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.912822 2557 scope.go:117] "RemoveContainer" containerID="5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50" Jul 22 13:26:09.913185 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:26:09.913165 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50\": container with ID starting with 5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50 not found: ID does not exist" containerID="5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50" Jul 22 13:26:09.913230 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.913192 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50"} err="failed to get container status \"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50\": rpc error: code = NotFound desc = could not find container \"5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50\": container with ID starting with 5f44adc491e8ce64479e49d90ecfb05d8aee9704cf0d4103f7a10f1e3af43b50 not found: ID does not exist" Jul 22 13:26:09.935186 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.935148 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:26:09.937792 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:09.937759 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/rate-test-model-d24327-kserve-57547677dc-vbk4v"] Jul 22 13:26:11.050854 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.050825 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.073440 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.073414 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089130910608_2e0df439-5f29-44ee-b26b-d8623bdb412d/test-curl-7777-140089130910608/0.log" Jul 22 13:26:11.086741 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.086701 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqnzz\" (UniqueName: \"kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz\") pod \"2e0df439-5f29-44ee-b26b-d8623bdb412d\" (UID: \"2e0df439-5f29-44ee-b26b-d8623bdb412d\") " Jul 22 13:26:11.088685 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.088661 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz" (OuterVolumeSpecName: "kube-api-access-pqnzz") pod "2e0df439-5f29-44ee-b26b-d8623bdb412d" (UID: "2e0df439-5f29-44ee-b26b-d8623bdb412d"). InnerVolumeSpecName "kube-api-access-pqnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:11.099871 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.099848 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:11.103874 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.103854 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:11.188459 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.188359 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqnzz\" (UniqueName: \"kubernetes.io/projected/2e0df439-5f29-44ee-b26b-d8623bdb412d-kube-api-access-pqnzz\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:11.228876 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.228839 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:11.229166 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229154 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b534030d-b318-43ed-8599-e0fbf719a7ca" containerName="main" Jul 22 13:26:11.229209 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229169 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b534030d-b318-43ed-8599-e0fbf719a7ca" containerName="main" Jul 22 13:26:11.229209 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229193 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e0df439-5f29-44ee-b26b-d8623bdb412d" containerName="test-curl-7777-140089130910608" Jul 22 13:26:11.229209 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229200 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e0df439-5f29-44ee-b26b-d8623bdb412d" containerName="test-curl-7777-140089130910608" Jul 22 13:26:11.229300 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229246 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e0df439-5f29-44ee-b26b-d8623bdb412d" containerName="test-curl-7777-140089130910608" Jul 22 13:26:11.229300 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.229255 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b534030d-b318-43ed-8599-e0fbf719a7ca" containerName="main" Jul 22 13:26:11.232093 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.232077 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.237675 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.237647 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:11.289428 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.289388 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbsn\" (UniqueName: \"kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn\") pod \"test-curl-7777-140089130910608\" (UID: \"6cd2b020-78e4-4539-bf52-7991bacab1f6\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.390895 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.390854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbsn\" (UniqueName: \"kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn\") pod \"test-curl-7777-140089130910608\" (UID: \"6cd2b020-78e4-4539-bf52-7991bacab1f6\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.399895 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.399867 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbsn\" (UniqueName: \"kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn\") pod \"test-curl-7777-140089130910608\" (UID: \"6cd2b020-78e4-4539-bf52-7991bacab1f6\") " pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.485385 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.485352 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e0df439-5f29-44ee-b26b-d8623bdb412d" path="/var/lib/kubelet/pods/2e0df439-5f29-44ee-b26b-d8623bdb412d/volumes" Jul 22 13:26:11.485777 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.485763 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b534030d-b318-43ed-8599-e0fbf719a7ca" path="/var/lib/kubelet/pods/b534030d-b318-43ed-8599-e0fbf719a7ca/volumes" Jul 22 13:26:11.543642 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.543604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:11.680415 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.680377 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:11.682876 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:11.682850 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd2b020_78e4_4539_bf52_7991bacab1f6.slice/crio-0b8842c8e424cc48ae06dc55aea24db4874850aab62bc27d97f6e9349d98beb3 WatchSource:0}: Error finding container 0b8842c8e424cc48ae06dc55aea24db4874850aab62bc27d97f6e9349d98beb3: Status 404 returned error can't find the container with id 0b8842c8e424cc48ae06dc55aea24db4874850aab62bc27d97f6e9349d98beb3 Jul 22 13:26:11.916698 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.916605 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089130910608" event={"ID":"6cd2b020-78e4-4539-bf52-7991bacab1f6","Type":"ContainerStarted","Data":"0b8842c8e424cc48ae06dc55aea24db4874850aab62bc27d97f6e9349d98beb3"} Jul 22 13:26:11.917800 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.917779 2557 scope.go:117] "RemoveContainer" containerID="551ebfde4327b814dd1769bd1f7c8e6a31be565d79cccb35117ec71f80e19d0d" Jul 22 13:26:11.917921 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:11.917781 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:12.923384 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:12.923344 2557 generic.go:356] "Generic (PLEG): container finished" podID="6cd2b020-78e4-4539-bf52-7991bacab1f6" containerID="40d6fa704cf5d7940c32c65b556f22e2d1a11221e18a87d9fe658d435e1d2b4d" exitCode=0 Jul 22 13:26:12.923840 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:12.923426 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-140089130910608" event={"ID":"6cd2b020-78e4-4539-bf52-7991bacab1f6","Type":"ContainerDied","Data":"40d6fa704cf5d7940c32c65b556f22e2d1a11221e18a87d9fe658d435e1d2b4d"} Jul 22 13:26:14.063445 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.063424 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:14.087651 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.087630 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7777-140089130910608_6cd2b020-78e4-4539-bf52-7991bacab1f6/test-curl-7777-140089130910608/0.log" Jul 22 13:26:14.113964 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.113937 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:14.117437 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.117411 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbsn\" (UniqueName: \"kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn\") pod \"6cd2b020-78e4-4539-bf52-7991bacab1f6\" (UID: \"6cd2b020-78e4-4539-bf52-7991bacab1f6\") " Jul 22 13:26:14.119398 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.119370 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-140089130910608"] Jul 22 13:26:14.119626 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.119606 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn" (OuterVolumeSpecName: "kube-api-access-tlbsn") pod "6cd2b020-78e4-4539-bf52-7991bacab1f6" (UID: "6cd2b020-78e4-4539-bf52-7991bacab1f6"). InnerVolumeSpecName "kube-api-access-tlbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:14.218725 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.218679 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbsn\" (UniqueName: \"kubernetes.io/projected/6cd2b020-78e4-4539-bf52-7991bacab1f6-kube-api-access-tlbsn\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:14.313804 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.313775 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2"] Jul 22 13:26:14.314117 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.314105 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd2b020-78e4-4539-bf52-7991bacab1f6" containerName="test-curl-7777-140089130910608" Jul 22 13:26:14.314155 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.314119 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd2b020-78e4-4539-bf52-7991bacab1f6" containerName="test-curl-7777-140089130910608" Jul 22 13:26:14.314186 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.314172 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd2b020-78e4-4539-bf52-7991bacab1f6" containerName="test-curl-7777-140089130910608" Jul 22 13:26:14.316928 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.316912 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:14.320157 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.320138 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:26:14.320257 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.320169 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:26:14.326357 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.326332 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2"] Jul 22 13:26:14.420601 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.420572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2\" (UID: \"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:14.521286 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.521198 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2\" (UID: \"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:14.521832 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.521812 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2\" (UID: \"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:14.627411 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.627372 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:14.762146 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.762122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2"] Jul 22 13:26:14.764001 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:14.763975 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3396ce79_5fd1_4139_9d3e_e5a1c4e15a70.slice/crio-e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32 WatchSource:0}: Error finding container e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32: Status 404 returned error can't find the container with id e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32 Jul 22 13:26:14.892515 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.892478 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:14.934969 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.934947 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:14.935092 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.934959 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8842c8e424cc48ae06dc55aea24db4874850aab62bc27d97f6e9349d98beb3" Jul 22 13:26:14.936520 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.936489 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" event={"ID":"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70","Type":"ContainerStarted","Data":"99c0647670e9b341589bc0500b335fb6026d12bcee02906462c4a08e27c2d9a5"} Jul 22 13:26:14.936623 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.936547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" event={"ID":"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70","Type":"ContainerStarted","Data":"e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32"} Jul 22 13:26:14.951023 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.950959 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" podStartSLOduration=0.950941637 podStartE2EDuration="950.941637ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:26:14.950368162 +0000 UTC m=+2714.037857360" watchObservedRunningTime="2026-07-22 13:26:14.950941637 +0000 UTC m=+2714.038430817" Jul 22 13:26:14.952507 ip-10-0-137-115 kubenswrapper[2557]: E0722 13:26:14.952480 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-140089130910608\" is forbidden: User \"system:node:ip-10-0-137-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-115.ec2.internal' and this object" podUID="6cd2b020-78e4-4539-bf52-7991bacab1f6" pod="opendatahub/test-curl-7777-140089130910608" Jul 22 13:26:14.994573 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:14.994545 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:15.486299 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:15.486270 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd2b020-78e4-4539-bf52-7991bacab1f6" path="/var/lib/kubelet/pods/6cd2b020-78e4-4539-bf52-7991bacab1f6/volumes" Jul 22 13:26:15.943190 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:15.943158 2557 generic.go:356] "Generic (PLEG): container finished" podID="3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" containerID="99c0647670e9b341589bc0500b335fb6026d12bcee02906462c4a08e27c2d9a5" exitCode=0 Jul 22 13:26:15.943379 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:15.943245 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" event={"ID":"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70","Type":"ContainerDied","Data":"99c0647670e9b341589bc0500b335fb6026d12bcee02906462c4a08e27c2d9a5"} Jul 22 13:26:17.078613 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.078587 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:17.146513 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.146477 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca\") pod \"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70\" (UID: \"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70\") " Jul 22 13:26:17.146875 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.146852 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" (UID: "3396ce79-5fd1-4139-9d3e-e5a1c4e15a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:26:17.248159 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.248073 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:17.955280 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.955241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" event={"ID":"3396ce79-5fd1-4139-9d3e-e5a1c4e15a70","Type":"ContainerDied","Data":"e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32"} Jul 22 13:26:17.955458 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.955301 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3dd0801f7cb2558d039236fcc835b1a0edeaa5eb1a727084eb95ccc4e787a32" Jul 22 13:26:17.955458 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:17.955253 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2" Jul 22 13:26:24.965689 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:24.965654 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:29.433520 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:29.433486 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:29.448831 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:29.448795 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2"] Jul 22 13:26:29.451536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:29.451510 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24327-jsgz2"] Jul 22 13:26:29.485033 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:29.484999 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" path="/var/lib/kubelet/pods/3396ce79-5fd1-4139-9d3e-e5a1c4e15a70/volumes" Jul 22 13:26:29.851565 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:29.851522 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:36.236587 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.236539 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:36.336498 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.336459 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:36.737115 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.737079 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:36.801488 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.801457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:36.897443 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.897411 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr"] Jul 22 13:26:36.897789 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.897776 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" containerName="revoke-keys" Jul 22 13:26:36.897850 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.897790 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" containerName="revoke-keys" Jul 22 13:26:36.897886 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.897857 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3396ce79-5fd1-4139-9d3e-e5a1c4e15a70" containerName="revoke-keys" Jul 22 13:26:36.903032 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.903014 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:36.906099 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.906079 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:26:36.906429 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.906415 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 13:26:36.913777 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:36.913755 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr"] Jul 22 13:26:37.024536 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.024440 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr\" (UID: \"7991e139-fca4-4a15-a183-85dbebfca25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:37.125125 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.125090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr\" (UID: \"7991e139-fca4-4a15-a183-85dbebfca25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:37.125688 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.125668 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr\" (UID: \"7991e139-fca4-4a15-a183-85dbebfca25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:37.213329 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.213291 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:37.350359 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.350333 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr"] Jul 22 13:26:37.352943 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:26:37.352911 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7991e139_fca4_4a15_a183_85dbebfca25d.slice/crio-4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732 WatchSource:0}: Error finding container 4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732: Status 404 returned error can't find the container with id 4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732 Jul 22 13:26:37.461343 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.461311 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:37.555395 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:37.555322 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:38.046480 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:38.046445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" event={"ID":"7991e139-fca4-4a15-a183-85dbebfca25d","Type":"ContainerStarted","Data":"7ff19cd51ca4360bc3d18cbdb314593fb0854babce974c92b39216720297e425"} Jul 22 13:26:38.046673 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:38.046495 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" event={"ID":"7991e139-fca4-4a15-a183-85dbebfca25d","Type":"ContainerStarted","Data":"4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732"} Jul 22 13:26:38.072752 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:38.072677 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" podStartSLOduration=2.072656084 podStartE2EDuration="2.072656084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:26:38.070788814 +0000 UTC m=+2737.158277996" watchObservedRunningTime="2026-07-22 13:26:38.072656084 +0000 UTC m=+2737.160145261" Jul 22 13:26:39.052727 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:39.052679 2557 generic.go:356] "Generic (PLEG): container finished" podID="7991e139-fca4-4a15-a183-85dbebfca25d" containerID="7ff19cd51ca4360bc3d18cbdb314593fb0854babce974c92b39216720297e425" exitCode=0 Jul 22 13:26:39.053134 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:39.052746 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" event={"ID":"7991e139-fca4-4a15-a183-85dbebfca25d","Type":"ContainerDied","Data":"7ff19cd51ca4360bc3d18cbdb314593fb0854babce974c92b39216720297e425"} Jul 22 13:26:40.193244 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:40.193218 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:40.355491 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:40.355390 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca\") pod \"7991e139-fca4-4a15-a183-85dbebfca25d\" (UID: \"7991e139-fca4-4a15-a183-85dbebfca25d\") " Jul 22 13:26:40.355721 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:40.355682 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7991e139-fca4-4a15-a183-85dbebfca25d" (UID: "7991e139-fca4-4a15-a183-85dbebfca25d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:26:40.456455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:40.456405 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7991e139-fca4-4a15-a183-85dbebfca25d-maas-api-service-ca\") on node \"ip-10-0-137-115.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.063813 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:41.063769 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" event={"ID":"7991e139-fca4-4a15-a183-85dbebfca25d","Type":"ContainerDied","Data":"4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732"} Jul 22 13:26:41.063990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:41.063824 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bac91143ebad7811486e6d51669403c743b228ff30930d446d3b59630ee1732" Jul 22 13:26:41.063990 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:41.063788 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr" Jul 22 13:26:47.490922 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:47.490885 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:51.977213 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:51.977173 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr"] Jul 22 13:26:51.979816 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:51.979790 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ffc1de-5nkdr"] Jul 22 13:26:51.980696 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:51.980670 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:52.367192 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:52.367109 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:53.485767 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:53.485737 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7991e139-fca4-4a15-a183-85dbebfca25d" path="/var/lib/kubelet/pods/7991e139-fca4-4a15-a183-85dbebfca25d/volumes" Jul 22 13:26:58.580431 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:58.580385 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:58.682439 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:58.682400 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:58.893741 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:58.893632 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:59.108500 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:59.108462 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:26:59.568234 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:26:59.568200 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c859549df-g727n_2be1a884-af4b-4ded-9107-15a956f2c509/authorino/0.log" Jul 22 13:27:05.117903 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:05.117861 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:27:05.819765 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:05.819733 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j9tp8_ad574278-7887-4d95-a7ba-06b85ca52bb3/manager/0.log" Jul 22 13:27:06.229822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:06.229795 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-w8dv8_0e24bdfe-a9c0-4ecd-a07d-c658438ee151/copy-manifests/0.log" Jul 22 13:27:06.237243 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:06.237220 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-w8dv8_0e24bdfe-a9c0-4ecd-a07d-c658438ee151/manager/0.log" Jul 22 13:27:06.826516 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:06.826485 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745435-j6cxf_84fc0dfe-d6ae-4e9b-8cdf-2be7d135fd4e/cleanup/0.log" Jul 22 13:27:08.306419 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:08.306377 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c859549df-g727n_2be1a884-af4b-4ded-9107-15a956f2c509/authorino/0.log" Jul 22 13:27:08.677249 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:08.677167 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qvqx7_1b205d99-3af7-4f56-930f-fe6f768d146e/kuadrant-console-plugin/0.log" Jul 22 13:27:09.611149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:09.611119 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dsbl7_9a166b8e-6365-4bc8-8c4e-37895d574c38/istio-proxy/0.log" Jul 22 13:27:09.981064 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:09.981027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-js4b9_b4ee986d-c002-40f9-afed-d233754a238e/kube-auth-proxy/0.log" Jul 22 13:27:10.118731 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:10.118683 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/1.log" Jul 22 13:27:10.847316 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:10.847283 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-dfqzm_a643b935-b0ae-476c-a0da-424e82a615f7/main/0.log" Jul 22 13:27:11.350319 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:11.350293 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-s525t_d44a70c1-d522-4be1-9238-c3087ab8ecaf/main/0.log" Jul 22 13:27:16.929019 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.928978 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2jzv/must-gather-5w5sr"] Jul 22 13:27:16.934966 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.934917 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7991e139-fca4-4a15-a183-85dbebfca25d" containerName="revoke-keys" Jul 22 13:27:16.934966 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.934965 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7991e139-fca4-4a15-a183-85dbebfca25d" containerName="revoke-keys" Jul 22 13:27:16.935346 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.935322 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7991e139-fca4-4a15-a183-85dbebfca25d" containerName="revoke-keys" Jul 22 13:27:16.939016 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.938990 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:16.941124 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.941096 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/must-gather-5w5sr"] Jul 22 13:27:16.942178 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.942153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g2jzv\"/\"default-dockercfg-5xtwz\"" Jul 22 13:27:16.943218 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.943195 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"kube-root-ca.crt\"" Jul 22 13:27:16.943352 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.943218 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"openshift-service-ca.crt\"" Jul 22 13:27:16.965953 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:16.965915 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:27:17.062063 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.062035 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024 Jul 22 13:27:17.082081 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.082044 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zqx\" (UniqueName: \"kubernetes.io/projected/2e234236-f23b-41d9-b5cc-92b80df4ea4f-kube-api-access-89zqx\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.082081 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.082081 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e234236-f23b-41d9-b5cc-92b80df4ea4f-must-gather-output\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.183182 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.183076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zqx\" (UniqueName: \"kubernetes.io/projected/2e234236-f23b-41d9-b5cc-92b80df4ea4f-kube-api-access-89zqx\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.183182 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.183118 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e234236-f23b-41d9-b5cc-92b80df4ea4f-must-gather-output\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.183472 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.183456 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e234236-f23b-41d9-b5cc-92b80df4ea4f-must-gather-output\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.193179 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.193146 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zqx\" (UniqueName: \"kubernetes.io/projected/2e234236-f23b-41d9-b5cc-92b80df4ea4f-kube-api-access-89zqx\") pod \"must-gather-5w5sr\" (UID: \"2e234236-f23b-41d9-b5cc-92b80df4ea4f\") " pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.249260 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.249219 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" Jul 22 13:27:17.389899 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:17.389864 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/must-gather-5w5sr"] Jul 22 13:27:17.390934 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:27:17.390911 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e234236_f23b_41d9_b5cc_92b80df4ea4f.slice/crio-0b0047bce249cced0d8cc4aef552488aba319f955d0f083cf5b3b71bee771650 WatchSource:0}: Error finding container 0b0047bce249cced0d8cc4aef552488aba319f955d0f083cf5b3b71bee771650: Status 404 returned error can't find the container with id 0b0047bce249cced0d8cc4aef552488aba319f955d0f083cf5b3b71bee771650 Jul 22 13:27:18.243781 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:18.243744 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" event={"ID":"2e234236-f23b-41d9-b5cc-92b80df4ea4f","Type":"ContainerStarted","Data":"2288aa6b7850d37e075a03a4d8c663ed6100a3642801c49ba8614300b0ffd839"} Jul 22 13:27:18.244130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:18.243798 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" event={"ID":"2e234236-f23b-41d9-b5cc-92b80df4ea4f","Type":"ContainerStarted","Data":"0b0047bce249cced0d8cc4aef552488aba319f955d0f083cf5b3b71bee771650"} Jul 22 13:27:19.253655 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:19.253607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" event={"ID":"2e234236-f23b-41d9-b5cc-92b80df4ea4f","Type":"ContainerStarted","Data":"c395b165d51a65fcf7d7d6b25aa3804ab9c80467c24e09f6d253c20c7c590494"} Jul 22 13:27:19.272014 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:19.271936 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2jzv/must-gather-5w5sr" podStartSLOduration=2.574042952 podStartE2EDuration="3.271914744s" totalImagesPullingTime="697.871792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:27:19.271331297 +0000 UTC m=+2778.358820477" watchObservedRunningTime="2026-07-22 13:27:19.271914744 +0000 UTC m=+2778.359403924" Jul 22 13:27:19.778931 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:19.778881 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jxnk6_bf7e899a-13ae-4fa5-a1be-4677dbf43f4a/global-pull-secret-syncer/0.log" Jul 22 13:27:19.839271 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:19.839232 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" Jul 22 13:27:19.945273 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:19.945240 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-115.ec2.internal_6c25f5faf03a7c576edb1dfa593890c7/haproxy/0.log" Jul 22 13:27:25.510486 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:25.510319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c859549df-g727n_2be1a884-af4b-4ded-9107-15a956f2c509/authorino/0.log" Jul 22 13:27:25.595011 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:25.594983 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qvqx7_1b205d99-3af7-4f56-930f-fe6f768d146e/kuadrant-console-plugin/0.log" Jul 22 13:27:27.994612 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:27.994529 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvskl_29d46ac7-b021-4a53-af43-1558cdd10349/node-exporter/0.log" Jul 22 13:27:28.017842 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.017807 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvskl_29d46ac7-b021-4a53-af43-1558cdd10349/kube-rbac-proxy/0.log" Jul 22 13:27:28.039537 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.039495 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvskl_29d46ac7-b021-4a53-af43-1558cdd10349/init-textfile/0.log" Jul 22 13:27:28.149799 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.149702 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/prometheus/0.log" Jul 22 13:27:28.171004 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.170977 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/config-reloader/0.log" Jul 22 13:27:28.204388 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.204353 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/thanos-sidecar/0.log" Jul 22 13:27:28.227384 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.227355 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/kube-rbac-proxy-web/0.log" Jul 22 13:27:28.255149 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.255106 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/kube-rbac-proxy/0.log" Jul 22 13:27:28.278743 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.278674 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/kube-rbac-proxy-thanos/0.log" Jul 22 13:27:28.304798 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.304771 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f6444703-53ba-42ad-8871-5f01350b043d/init-config-reloader/0.log" Jul 22 13:27:28.418435 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.418357 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67cc54c98f-zg7gk_0d97a441-7fe1-455f-a164-523e7a67be35/telemeter-client/0.log" Jul 22 13:27:28.439319 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.439292 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67cc54c98f-zg7gk_0d97a441-7fe1-455f-a164-523e7a67be35/reload/0.log" Jul 22 13:27:28.460927 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:28.460895 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67cc54c98f-zg7gk_0d97a441-7fe1-455f-a164-523e7a67be35/kube-rbac-proxy/0.log" Jul 22 13:27:29.501732 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.500794 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p"] Jul 22 13:27:29.509168 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.509140 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.515064 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.515029 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p"] Jul 22 13:27:29.607641 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.607599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-proc\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.607641 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.607639 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-sys\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.607904 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.607848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-lib-modules\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.607963 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.607935 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-podres\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.608130 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.608021 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98872\" (UniqueName: \"kubernetes.io/projected/8a3c3d02-acfe-4342-885f-ca9d87781af6-kube-api-access-98872\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.709577 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.709532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-podres\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.709914 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.709886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98872\" (UniqueName: \"kubernetes.io/projected/8a3c3d02-acfe-4342-885f-ca9d87781af6-kube-api-access-98872\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.710440 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.710421 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-proc\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.710651 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.710635 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-sys\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.710939 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.710921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-lib-modules\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.711240 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.710815 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-sys\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.711360 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.710588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-proc\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.711455 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.711153 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-lib-modules\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.711549 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.709701 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a3c3d02-acfe-4342-885f-ca9d87781af6-podres\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.719960 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.719929 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98872\" (UniqueName: \"kubernetes.io/projected/8a3c3d02-acfe-4342-885f-ca9d87781af6-kube-api-access-98872\") pod \"perf-node-gather-daemonset-l549p\" (UID: \"8a3c3d02-acfe-4342-885f-ca9d87781af6\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:29.816006 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.815922 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-sdlx9_fcaea545-98bc-499d-b933-acaabd88910d/networking-console-plugin/0.log" Jul 22 13:27:29.835456 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:29.835421 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:30.026512 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:30.026482 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p"] Jul 22 13:27:30.029118 ip-10-0-137-115 kubenswrapper[2557]: W0722 13:27:30.029081 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a3c3d02_acfe_4342_885f_ca9d87781af6.slice/crio-4d997cb04d7349d65c8336806124d69556e21a8be09db0661f3f3d34cea9007c WatchSource:0}: Error finding container 4d997cb04d7349d65c8336806124d69556e21a8be09db0661f3f3d34cea9007c: Status 404 returned error can't find the container with id 4d997cb04d7349d65c8336806124d69556e21a8be09db0661f3f3d34cea9007c Jul 22 13:27:30.356886 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:30.356797 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" event={"ID":"8a3c3d02-acfe-4342-885f-ca9d87781af6","Type":"ContainerStarted","Data":"3efa759539523c828eba2b8a9a472a2e446a1ad8bfb9fe57c645a301db6d4af9"} Jul 22 13:27:30.356886 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:30.356854 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" event={"ID":"8a3c3d02-acfe-4342-885f-ca9d87781af6","Type":"ContainerStarted","Data":"4d997cb04d7349d65c8336806124d69556e21a8be09db0661f3f3d34cea9007c"} Jul 22 13:27:30.357112 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:30.356909 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:30.378620 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:30.378552 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" podStartSLOduration=1.378529406 podStartE2EDuration="1.378529406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 13:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:27:30.375813257 +0000 UTC m=+2789.463302437" watchObservedRunningTime="2026-07-22 13:27:30.378529406 +0000 UTC m=+2789.466018587" Jul 22 13:27:31.025988 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:31.025957 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-g59c5_529542b2-c375-4db7-a47d-a91fc2eb41a8/download-server/0.log" Jul 22 13:27:32.642802 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:32.642772 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gk4nb_ca97e449-ed09-4e0b-81af-871d3951b7ca/dns/0.log" Jul 22 13:27:32.667170 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:32.667142 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gk4nb_ca97e449-ed09-4e0b-81af-871d3951b7ca/kube-rbac-proxy/0.log" Jul 22 13:27:32.860822 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:32.860796 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z7jhj_1b69eb66-ef86-415a-be83-810999d5cd1d/dns-node-resolver/0.log" Jul 22 13:27:33.422433 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:33.422375 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cf68bf6f4-l8crz_8f1a1005-c4b8-47e2-912d-d5ede38699aa/registry/0.log" Jul 22 13:27:33.476174 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:33.476139 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qx62d_70499e3b-ae4c-41d1-9549-0139ecf082e3/node-ca/0.log" Jul 22 13:27:34.480991 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:34.480948 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dsbl7_9a166b8e-6365-4bc8-8c4e-37895d574c38/istio-proxy/0.log" Jul 22 13:27:34.652213 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:34.652187 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-js4b9_b4ee986d-c002-40f9-afed-d233754a238e/kube-auth-proxy/0.log" Jul 22 13:27:34.688072 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:34.688039 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/0.log" Jul 22 13:27:34.721701 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:34.721656 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/1.log" Jul 22 13:27:35.516265 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:35.516242 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jglnd_a71bf077-f306-4681-8f97-614a73c15e41/serve-healthcheck-canary/0.log" Jul 22 13:27:36.021838 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:36.021804 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx4xd_8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2/kube-rbac-proxy/0.log" Jul 22 13:27:36.043277 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:36.043247 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx4xd_8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2/exporter/0.log" Jul 22 13:27:36.066729 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:36.066683 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cx4xd_8f43a315-e3a6-4ed9-9c3d-fbcf275fe4e2/extractor/0.log" Jul 22 13:27:36.373426 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:36.373344 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-l549p" Jul 22 13:27:39.099289 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:39.099249 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j9tp8_ad574278-7887-4d95-a7ba-06b85ca52bb3/manager/0.log" Jul 22 13:27:39.276535 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:39.276507 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-w8dv8_0e24bdfe-a9c0-4ecd-a07d-c658438ee151/manager/0.log" Jul 22 13:27:39.301865 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:39.301833 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-w8dv8_0e24bdfe-a9c0-4ecd-a07d-c658438ee151/copy-manifests/0.log" Jul 22 13:27:40.267156 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.267117 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dsbl7_9a166b8e-6365-4bc8-8c4e-37895d574c38/istio-proxy/0.log" Jul 22 13:27:40.406647 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.406613 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-js4b9_b4ee986d-c002-40f9-afed-d233754a238e/kube-auth-proxy/0.log" Jul 22 13:27:40.427674 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.427634 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/0.log" Jul 22 13:27:40.459750 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.459696 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/1.log" Jul 22 13:27:40.644736 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.644639 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8b6495d78-2lvwh_3befb592-c4a5-48f6-8194-9ce298b5ccd8/manager/0.log" Jul 22 13:27:40.691694 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:40.691658 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jklkh_10654824-7f2a-4b90-87a2-7f91b422a3d1/openshift-lws-operator/0.log" Jul 22 13:27:41.721565 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:41.721519 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4dsbl7_9a166b8e-6365-4bc8-8c4e-37895d574c38/istio-proxy/0.log" Jul 22 13:27:41.875627 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:41.875603 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-js4b9_b4ee986d-c002-40f9-afed-d233754a238e/kube-auth-proxy/0.log" Jul 22 13:27:41.901481 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:41.900095 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/0.log" Jul 22 13:27:41.936425 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:41.936372 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-t2mvg_18a6ea09-f03e-4b79-8f3b-b26007b967de/istio-proxy/1.log" Jul 22 13:27:42.705771 ip-10-0-137-115 kubenswrapper[2557]: I0722 13:27:42.705731 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bqh6h_1dd88ee1-0a1a-4206-b382-0785f8ba1fd0/konnectivity-agent/0.log" limit=1024