Jul 06 19:30:34.229395 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:30:34.229410 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:30:34.229420 ip-10-0-136-184 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:30:34.229773 ip-10-0-136-184 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:30:44.377664 ip-10-0-136-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:30:44.377684 ip-10-0-136-184 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8be3a60c869f42e689e205d5e7b2634b -- Jul 06 19:32:43.205364 ip-10-0-136-184 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:32:45.013705 ip-10-0-136-184 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.611358 2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615693 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615707 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615717 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615721 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615726 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615730 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.024876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615733 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615736 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615738 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615741 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615744 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615746 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615749 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615752 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615755 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615758 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615768 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615770 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615773 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615776 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615778 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615781 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615784 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615787 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615789 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615792 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.026002 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615794 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615797 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615800 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615802 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615805 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615807 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615810 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615813 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615815 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615817 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615820 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615825 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615829 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615832 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615835 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615838 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615840 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615843 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615847 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.026703 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615849 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615853 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615855 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615858 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615860 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615863 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615865 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615868 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615871 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615873 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615876 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615879 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615882 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615886 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615888 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615891 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615893 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615896 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615898 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615901 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.027909 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615903 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615906 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615908 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615911 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615913 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615916 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615919 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615922 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615924 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615927 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615930 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615933 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615935 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615938 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615941 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615944 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615946 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615949 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615951 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.029069 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615954 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615960 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615963 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615966 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615969 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615972 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615975 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615978 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615981 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615984 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615986 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615989 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615992 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615995 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.615997 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616741 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616748 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616752 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616755 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616758 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.029961 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616762 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616765 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616768 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616771 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616774 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616777 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616780 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616783 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616786 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616788 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616791 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616793 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616796 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616798 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616802 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616805 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616808 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616811 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616813 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616816 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.031111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616819 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616821 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616825 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616828 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616831 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616834 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616836 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616839 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616842 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616844 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616847 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616850 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616854 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616858 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616862 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616865 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616868 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616871 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616874 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.032250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616876 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616899 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616918 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616922 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616925 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616929 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616932 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616935 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616938 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616941 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616944 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616947 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616951 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616954 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616956 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616960 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616963 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616965 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616968 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616971 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.033380 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616973 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616976 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616979 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616981 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616984 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616986 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616989 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616991 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.616994 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617000 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617002 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617005 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617007 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617010 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617014 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617016 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617021 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617025 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617029 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.034454 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617031 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617034 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617037 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617040 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617043 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617046 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617049 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617052 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617056 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617074 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617078 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617081 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617084 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617087 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617089 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617092 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617535 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617542 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617545 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617548 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.035461 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617551 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617554 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617556 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617559 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617562 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617564 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617567 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617570 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617574 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617578 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617581 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617587 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617590 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617593 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617597 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617599 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617602 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617605 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617607 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617610 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.036652 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617613 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617615 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617618 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617621 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617624 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617626 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617629 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617632 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617634 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617637 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617639 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617642 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617644 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617647 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617649 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617652 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617655 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617658 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617660 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617663 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.037781 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617666 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617669 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617671 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617675 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617678 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617681 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617683 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617686 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617689 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617691 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617694 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617696 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617699 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617701 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617703 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617707 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617709 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617712 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617715 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617718 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.039560 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617720 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617723 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617725 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617729 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617731 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617734 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617736 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617739 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617744 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617747 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617749 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617752 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617754 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617757 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617760 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617764 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617766 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617769 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617772 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.040695 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617774 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617776 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617779 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617781 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617786 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617790 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617794 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617797 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617800 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617803 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617805 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617808 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617811 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617814 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617817 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.617820 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617885 2584 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617892 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617899 2584 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617903 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617908 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:32:45.042247 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617911 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617915 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617919 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617922 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617925 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617929 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617932 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617937 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617940 2584 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617943 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617946 2584 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617949 2584 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617953 2584 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617957 2584 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617960 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617964 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617967 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617970 2584 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617974 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617977 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617980 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617984 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617987 2584 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617990 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:32:45.043376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617993 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617996 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.617999 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618004 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618007 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618009 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618012 2584 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618016 2584 flags.go:64] FLAG: --enable-server="true" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618019 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618024 2584 flags.go:64] FLAG: --event-burst="100" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618027 2584 flags.go:64] FLAG: --event-qps="50" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618029 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618033 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618036 2584 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618040 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618045 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618048 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618051 2584 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618054 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618057 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618076 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618079 2584 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618082 2584 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618086 2584 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618089 2584 flags.go:64] FLAG: --feature-gates="" Jul 06 19:32:45.045481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618093 2584 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618096 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618099 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618102 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618105 2584 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618108 2584 flags.go:64] FLAG: --help="false" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618111 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618114 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618118 2584 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618120 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618124 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618127 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618131 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618134 2584 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618137 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618140 2584 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618143 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618146 2584 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618149 2584 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618152 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618155 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618158 2584 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618163 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618171 2584 flags.go:64] FLAG: --lock-file="" Jul 06 19:32:45.046510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618175 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618178 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618181 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618191 2584 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618195 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618200 2584 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618206 2584 flags.go:64] FLAG: --logging-format="text" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618211 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618215 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618218 2584 flags.go:64] FLAG: --manifest-url="" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618221 2584 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618225 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618228 2584 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618233 2584 flags.go:64] FLAG: --max-pods="110" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618236 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618239 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618242 2584 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618245 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618248 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618251 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618254 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618261 2584 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618264 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618267 2584 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:32:45.047388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618270 2584 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618273 2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618279 2584 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618282 2584 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618285 2584 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618288 2584 flags.go:64] FLAG: --port="10250" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618293 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618296 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a849014d22fa386" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618299 2584 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618303 2584 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618306 2584 flags.go:64] FLAG: --register-node="true" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618309 2584 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618312 2584 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618315 2584 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618318 2584 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618322 2584 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618326 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618329 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618332 2584 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618334 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618337 2584 flags.go:64] FLAG: --runonce="false" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618340 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618344 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618346 2584 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618349 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:32:45.048238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618352 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618355 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618359 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618362 2584 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618365 2584 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618368 2584 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618370 2584 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618373 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618376 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618379 2584 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618382 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618387 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618390 2584 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618394 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618398 2584 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618401 2584 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618404 2584 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618407 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618410 2584 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618413 2584 flags.go:64] FLAG: --v="2" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618417 2584 flags.go:64] FLAG: --version="false" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618421 2584 flags.go:64] FLAG: --vmodule="" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618426 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.618429 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618541 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.049543 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618545 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618548 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618550 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618553 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618556 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618559 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618561 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618564 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618567 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618570 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618572 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618575 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618586 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618589 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618592 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618595 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618597 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618600 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618603 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618606 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.050441 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618610 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618613 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618616 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618619 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618622 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618625 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618628 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618630 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618633 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618637 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618639 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618642 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618645 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618647 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618650 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618652 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618655 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618658 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618660 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618663 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.051611 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618665 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618668 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618671 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618673 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618676 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618679 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618683 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618686 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618689 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618692 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618695 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618698 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618701 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618704 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618707 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618709 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618712 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618715 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618717 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618720 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.052335 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618722 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618726 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618729 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618732 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618736 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618739 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618742 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618745 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618749 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618752 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618755 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618758 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618761 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618763 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618766 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618768 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618771 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618774 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618776 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.053604 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618779 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618781 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618784 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618787 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618789 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618793 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618795 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618798 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618801 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618803 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618806 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618808 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618811 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618813 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618817 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618820 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618822 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618825 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.054203 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.618828 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.619833 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.627787 2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.627806 2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.627826 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.627836 2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627910 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627916 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627920 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627923 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627926 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627928 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627931 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627934 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627937 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.055632 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627939 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627942 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627945 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627947 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627950 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627953 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627955 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627959 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627963 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627966 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627969 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627972 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627975 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627977 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627980 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627984 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627986 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627989 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627992 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627995 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.249770 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.627997 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628000 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628003 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628006 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628008 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628011 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628014 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628016 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628019 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628021 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628024 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628026 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628029 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628032 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628034 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628037 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628039 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628042 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628045 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628047 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.271621 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628050 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628053 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628056 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628058 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628075 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628078 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628081 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628084 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628087 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628090 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628092 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628095 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628097 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628100 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628104 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628108 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628112 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628115 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.272300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628118 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628121 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628125 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628128 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628130 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628133 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628136 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628139 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628142 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628144 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628147 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628149 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628152 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628155 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628158 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628161 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628164 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628167 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628169 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628172 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.273390 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628175 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628178 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628180 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628183 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628186 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628188 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628191 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628194 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628196 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628199 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628202 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628204 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.628209 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628320 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628324 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:32:45.274021 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628328 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628332 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628335 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628337 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628343 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628346 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628349 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628351 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628354 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628356 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628362 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628365 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628367 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628370 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628373 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628375 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628378 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628380 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628383 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628385 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:32:45.274634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628388 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628391 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628393 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628396 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628398 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628401 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628403 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628406 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628409 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628411 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628414 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628416 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628419 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628421 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628424 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628426 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628430 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628433 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628435 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628438 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:32:45.275250 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628440 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628443 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628447 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628450 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628452 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628455 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628457 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628460 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628462 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628465 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628467 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628470 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628472 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628475 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628477 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628480 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628483 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628485 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628488 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628490 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:32:45.275876 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628493 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628495 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628498 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628500 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628503 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628505 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628508 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628510 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628514 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628517 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628520 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628522 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628525 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628528 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628532 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628536 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628539 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628542 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628545 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628548 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:32:45.276502 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628551 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628554 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628557 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628559 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628562 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628565 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628568 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628571 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628573 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628576 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628579 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628581 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628584 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628587 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628589 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628593 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:32:45.277330 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:43.628596 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.628601 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.628704 2584 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.631791 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.632915 2584 server.go:1019] "Starting client certificate rotation" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.633013 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:43.633049 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.962431 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.963549 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.978204 2584 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.981245 2584 log.go:25] "Validated CRI v1 image API" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.981961 2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:32:45.277969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.986577 2584 fs.go:135] Filesystem UUIDs: map[2878223b-7cd7-48f4-9b8b-8180c1d18bec:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d6627682-decd-4a6a-a427-b5785ba1a9fd:/dev/nvme0n1p4] Jul 06 19:32:45.278585 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.986595 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.993358 2584 manager.go:216] Machine: {Timestamp:2026-07-06 19:32:44.992266236 +0000 UTC m=+0.339909401 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099327 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2072cb030d718d6b267c58d404cf00 SystemUUID:ec2072cb-030d-718d-6b26-7c58d404cf00 BootID:8be3a60c-869f-42e6-89e2-05d5e7b2634b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:ab:03:de:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:ab:03:de:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:4a:35:8a:b5:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.993460 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.993579 2584 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.993827 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.996607 2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.996634 2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:32:45.278691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.997302 2584 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.997311 2584 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.997318 2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.997943 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:44.999211 2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.001814 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.001947 2584 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.001975 2584 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.001985 2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.001995 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.002005 2584 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.002013 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.002403 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.002426 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.004868 2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.007069 2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.007097 2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008167 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008180 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008187 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008193 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008202 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008208 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008216 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008222 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008229 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008235 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008243 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.008248 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.009052 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.009060 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.012364 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.012463 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.012459 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.012869 2584 server.go:1268] "Started kubelet" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.012914 2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.012941 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.013006 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.017674 2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:32:45.278998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.019291 2584 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.019401 2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.020477 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99wrn" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.021220 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-184.ec2.internal.18bfca42a2f85e99 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-184.ec2.internal,UID:ip-10-0-136-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-184.ec2.internal,},FirstTimestamp:2026-07-06 19:32:45.012844185 +0000 UTC m=+0.360487352,LastTimestamp:2026-07-06 19:32:45.012844185 +0000 UTC m=+0.360487352,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-184.ec2.internal,}" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.026642 2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.026644 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.026713 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027236 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027377 2584 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027395 2584 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027498 2584 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027505 2584 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027613 2584 factory.go:55] Registering systemd factory Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027629 2584 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027814 2584 factory.go:153] Registering CRI-O factory Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027826 2584 factory.go:223] Registration of the crio container factory successfully Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027886 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027910 2584 factory.go:103] Registering Raw factory Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.027941 2584 manager.go:1196] Started watching for new ooms in manager Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.028349 2584 manager.go:349] Starting recovery of all containers Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.029517 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-99wrn" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.039002 2584 manager.go:354] Recovery completed Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.044863 2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-184.ec2.internal\" not found" node="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.045410 2584 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.047194 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.052784 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.054566 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.054601 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.054616 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.056302 2584 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.056310 2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.056317 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.056361 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057087 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.280083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057111 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057122 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057398 2584 policy_none.go:49] "None policy: Start" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057414 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.057424 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.058211 2584 policy_none.go:47] "Start" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.090642 2584 manager.go:337] "Starting Device Plugin manager" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.090836 2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.090850 2584 server.go:87] "Starting device plugin registration server" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.091070 2584 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.091082 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.091168 2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.091250 2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.091260 2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.091814 2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.091869 2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.191320 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.192230 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.192259 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.192270 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.192297 2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.201963 2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.201984 2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-184.ec2.internal\": node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.215623 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.216784 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.216816 2584 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.216833 2584 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.216862 2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.217796 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.281033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.219110 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.317516 2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal"] Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.317618 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.317861 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.320051 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.320071 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.320080 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.322371 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.322731 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.322753 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323697 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323714 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323723 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323902 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323913 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.323963 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.326040 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.326063 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.326958 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.326986 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.327000 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.328815 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.328833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.328851 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c8ec2323b990d6957cedec6f5b7c41-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.ec2.internal\" (UID: \"a6c8ec2323b990d6957cedec6f5b7c41\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.365723 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.ec2.internal\" not found" node="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.371253 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-184.ec2.internal\" not found" node="ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.423305 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.418150 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429218 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429236 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c8ec2323b990d6957cedec6f5b7c41-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.ec2.internal\" (UID: \"a6c8ec2323b990d6957cedec6f5b7c41\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429281 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c8ec2323b990d6957cedec6f5b7c41-config\") pod \"kube-apiserver-proxy-ip-10-0-136-184.ec2.internal\" (UID: \"a6c8ec2323b990d6957cedec6f5b7c41\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429298 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.430344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.429309 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172e65fcdcf827162778b41b6338da26-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal\" (UID: \"172e65fcdcf827162778b41b6338da26\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.595780 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.518922 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.788282 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.619439 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:45.788282 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.668875 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.788282 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.675622 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:45.788282 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.720433 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.008361 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.820938 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.008361 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:45.921486 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.008361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.959898 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:32:46.008361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.960057 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:32:46.008361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:45.960081 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:32:46.220228 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:46.022098 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.220228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.033394 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:27:45 +0000 UTC" deadline="2028-02-18 22:03:05.314601854 +0000 UTC" Jul 06 19:32:46.220228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.033417 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14210h30m19.28118798s" Jul 06 19:32:46.220228 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:46.122750 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.398367 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:46.223653 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.398367 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:46.324054 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.420825 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:46.420785 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172e65fcdcf827162778b41b6338da26.slice/crio-7e7a02c6eb83bf826869a7613086f647672dc3e1d8dc59019d9571308908faad WatchSource:0}: Error finding container 7e7a02c6eb83bf826869a7613086f647672dc3e1d8dc59019d9571308908faad: Status 404 returned error can't find the container with id 7e7a02c6eb83bf826869a7613086f647672dc3e1d8dc59019d9571308908faad Jul 06 19:32:46.421050 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:46.421034 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c8ec2323b990d6957cedec6f5b7c41.slice/crio-fad26cd7f37ee96d6d3cd094ca35b86f79eb8c546b4c821fae9c33c5b525e613 WatchSource:0}: Error finding container fad26cd7f37ee96d6d3cd094ca35b86f79eb8c546b4c821fae9c33c5b525e613: Status 404 returned error can't find the container with id fad26cd7f37ee96d6d3cd094ca35b86f79eb8c546b4c821fae9c33c5b525e613 Jul 06 19:32:46.424119 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:46.424102 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-184.ec2.internal\" not found" Jul 06 19:32:46.424814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.424799 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:32:46.453011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.452987 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:46.460564 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.460547 2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:46.527968 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.527942 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" Jul 06 19:32:46.539616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.539596 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:32:46.540564 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.540552 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" Jul 06 19:32:46.552937 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:46.552912 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:32:47.004115 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.004085 2584 apiserver.go:52] "Watching apiserver" Jul 06 19:32:47.010450 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.010424 2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:32:47.010852 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.010817 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-89zgx","kube-system/konnectivity-agent-6z7rf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc","openshift-cluster-node-tuning-operator/tuned-xxzw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal","openshift-multus/multus-9749d","openshift-multus/multus-additional-cni-plugins-zjlqm","kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal","openshift-dns/node-resolver-qqkrj","openshift-image-registry/node-ca-r2rtw","openshift-multus/network-metrics-daemon-9vmzt","openshift-network-diagnostics/network-check-target-wpfsm","openshift-network-operator/iptables-alerter-d2jk7","openshift-ovn-kubernetes/ovnkube-node-9nzxn"] Jul 06 19:32:47.013362 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.013341 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.015470 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.015414 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.015674 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.015655 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.015869 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.015855 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:32:47.015937 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.015899 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9tzwk\"" Jul 06 19:32:47.016063 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.016047 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:32:47.016063 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.016058 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:32:47.016243 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.016227 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.016343 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.016320 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:47.017882 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.017861 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.019411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.019386 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:32:47.019652 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.019633 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.019652 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.019645 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-99zhn\"" Jul 06 19:32:47.019959 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.019940 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.020629 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.020226 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.022323 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.022302 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.022566 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.022550 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ft7qs\"" Jul 06 19:32:47.022753 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.022737 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.023056 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.023035 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.025105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.024665 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.025105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.024944 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:32:47.025105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025014 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:32:47.025105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025083 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:32:47.025105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025089 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7rn44\"" Jul 06 19:32:47.025634 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025407 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:32:47.025634 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025442 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9749d" Jul 06 19:32:47.025634 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.025486 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.027009 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.026990 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:32:47.027527 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.027508 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.028204 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.028171 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:32:47.028296 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.028202 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-658z8\"" Jul 06 19:32:47.029667 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.029452 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fwsv\"" Jul 06 19:32:47.029667 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.029542 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.029667 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.029458 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.030566 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.030250 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.031997 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.031917 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:32:47.032261 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.032242 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.032342 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.032259 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9j9dv\"" Jul 06 19:32:47.032487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.032467 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.033109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.033087 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.034666 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.034646 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:32:47.036881 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.036577 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dx8c6\"" Jul 06 19:32:47.036881 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.036749 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.037035 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.036940 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-slash\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.037035 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037025 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03881413-57f5-4fc2-bc7d-94ae5db6c711-hosts-file\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.037126 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037060 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qqh\" (UniqueName: \"kubernetes.io/projected/03881413-57f5-4fc2-bc7d-94ae5db6c711-kube-api-access-k4qqh\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.037126 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037093 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-bin\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.037213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037131 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cni-binary-copy\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.037213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037158 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-multus\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.037213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037182 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.037341 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037246 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-registration-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.037341 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037284 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-sys\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.037535 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037516 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-env-overrides\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.037589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037547 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-bin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.037589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037564 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-tuned\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.037677 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037647 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-systemd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.037725 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-kubelet\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037801 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-hostroot\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037837 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-os-release\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037871 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbqk\" (UniqueName: \"kubernetes.io/projected/7db4adcb-c8ea-4583-9473-cb91e59c5627-kube-api-access-4pbqk\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037939 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysconfig\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.037990 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-lib-modules\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038063 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-k8s-cni-cncf-io\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038109 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038109 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-etc-kubernetes\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038227 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038251 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038283 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-kubelet\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038300 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-netns\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038318 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-tmp\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038338 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-systemd-units\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038362 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038378 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-conf-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038402 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-systemd\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038420 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-etc-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038476 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-netd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038505 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdcf203-d48e-4833-8762-4ef8e039151f-ovn-node-metrics-cert\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038534 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-daemon-config\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038564 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-multus-certs\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038590 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-kubernetes\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038620 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-run\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038648 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-ovn\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038674 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-netns\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038715 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-socket-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038816 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:32:47.038988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038898 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p5h58\"" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.038998 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-etc-selinux\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039029 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-sys-fs\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039090 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-modprobe-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039128 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039175 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039179 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039251 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7db4adcb-c8ea-4583-9473-cb91e59c5627-iptables-alerter-script\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039288 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039301 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-var-lib-kubelet\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039333 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.039420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039389 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-script-lib\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039429 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-os-release\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039463 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgcw\" (UniqueName: \"kubernetes.io/projected/da0c2dfe-f30f-4cfe-825e-a580daea42ea-kube-api-access-nlgcw\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039596 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039632 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-host\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039662 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-var-lib-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039685 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz786\" (UniqueName: \"kubernetes.io/projected/9bdcf203-d48e-4833-8762-4ef8e039151f-kube-api-access-vz786\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039733 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.039916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.039798 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvx97\" (UniqueName: \"kubernetes.io/projected/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-kube-api-access-wvx97\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040391 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-log-socket\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040442 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-config\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040473 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-socket-dir-parent\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040532 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cnibin\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040558 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82jw\" (UniqueName: \"kubernetes.io/projected/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-kube-api-access-d82jw\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040592 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzm4\" (UniqueName: \"kubernetes.io/projected/05aac2b6-8cf9-415c-9394-e8e39a0b0948-kube-api-access-fkzm4\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040631 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040658 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-system-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040695 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cnibin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040717 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040741 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.040847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.040778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.041178 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-device-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041250 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/ef405f5e-43cb-441b-9f77-109cacbde554-kube-api-access-qsnkb\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041396 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-conf\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041445 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-node-log\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041470 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03881413-57f5-4fc2-bc7d-94ae5db6c711-tmp-dir\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.041557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.041492 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7db4adcb-c8ea-4583-9473-cb91e59c5627-host-slash\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.043216 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.043199 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:32:47.043832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.043778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:47.043949 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.043840 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:47.087751 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.087725 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrj6k" Jul 06 19:32:47.096087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.096064 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:47.096347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.096324 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrj6k" Jul 06 19:32:47.129363 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.129332 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:32:47.141864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141834 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-bin\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.141864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141870 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cni-binary-copy\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141887 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-multus\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141902 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141918 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-registration-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141952 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-sys\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.141977 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-env-overrides\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142000 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-bin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142030 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56edb1f5-00f9-4330-b62e-de016f1387af-konnectivity-ca\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142055 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83256f13-fd24-4c93-938d-c5ee1e8ecc93-host\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.142097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142080 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-kubelet-config\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142113 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-tuned\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-systemd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142166 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-kubelet\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142189 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-hostroot\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142211 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-os-release\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142235 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbqk\" (UniqueName: \"kubernetes.io/projected/7db4adcb-c8ea-4583-9473-cb91e59c5627-kube-api-access-4pbqk\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56edb1f5-00f9-4330-b62e-de016f1387af-agent-certs\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142292 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysconfig\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-lib-modules\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142338 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-k8s-cni-cncf-io\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142360 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-etc-kubernetes\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142407 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.142521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142452 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-kubelet\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142595 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-bin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142632 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-systemd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142666 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-netns\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142692 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-dbus\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142716 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142739 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-tmp\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142763 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-systemd-units\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142788 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142844 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-conf-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-systemd\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142899 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-etc-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142911 2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142996 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-sys\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143012 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-registration-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.143894 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143022 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143059 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-netd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143068 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-netns\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143101 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-hostroot\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143138 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-bin\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143175 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143213 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-conf-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143227 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143268 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-systemd\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143301 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-etc-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143317 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-k8s-cni-cncf-io\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143366 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysconfig\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143436 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-kubelet\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143473 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-lib-modules\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143481 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-var-lib-cni-multus\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.144624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143483 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-os-release\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143510 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-etc-kubernetes\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143103 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-kubelet\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.142946 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-cni-netd\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143678 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143681 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-systemd-units\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143728 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdcf203-d48e-4833-8762-4ef8e039151f-ovn-node-metrics-cert\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143766 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-daemon-config\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-multus-certs\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143886 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-kubernetes\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-run\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143940 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-env-overrides\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143948 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-ovn\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.143974 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-netns\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144001 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cni-binary-copy\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.145356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144016 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-netns\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144017 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-host-run-multus-certs\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144071 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-kubernetes\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144102 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-socket-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144117 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-run\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144128 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-etc-selinux\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144238 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-sys-fs\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144287 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-modprobe-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144313 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144356 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144389 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7db4adcb-c8ea-4583-9473-cb91e59c5627-iptables-alerter-script\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144414 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-var-lib-kubelet\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144391 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-run-ovn\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146073 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-script-lib\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144522 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-os-release\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144527 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-daemon-config\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144533 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144548 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgcw\" (UniqueName: \"kubernetes.io/projected/da0c2dfe-f30f-4cfe-825e-a580daea42ea-kube-api-access-nlgcw\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144573 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-host\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144605 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-etc-selinux\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144625 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-var-lib-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144651 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz786\" (UniqueName: \"kubernetes.io/projected/9bdcf203-d48e-4833-8762-4ef8e039151f-kube-api-access-vz786\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144677 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144706 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-socket-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144748 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvx97\" (UniqueName: \"kubernetes.io/projected/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-kube-api-access-wvx97\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144757 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-var-lib-openvswitch\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.146707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-log-socket\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144801 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-config\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-socket-dir-parent\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144850 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cnibin\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144876 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82jw\" (UniqueName: \"kubernetes.io/projected/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-kube-api-access-d82jw\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144902 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83256f13-fd24-4c93-938d-c5ee1e8ecc93-serviceca\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144947 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhj7\" (UniqueName: \"kubernetes.io/projected/83256f13-fd24-4c93-938d-c5ee1e8ecc93-kube-api-access-zkhj7\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144967 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.144976 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzm4\" (UniqueName: \"kubernetes.io/projected/05aac2b6-8cf9-415c-9394-e8e39a0b0948-kube-api-access-fkzm4\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145005 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145030 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-sys-fs\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145073 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-system-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145091 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-socket-dir-parent\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145092 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cnibin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.147375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145132 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145142 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-cnibin\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145165 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-device-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145223 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/ef405f5e-43cb-441b-9f77-109cacbde554-kube-api-access-qsnkb\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145253 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-conf\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145253 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-script-lib\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145248 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-multus-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-node-log\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03881413-57f5-4fc2-bc7d-94ae5db6c711-tmp-dir\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7db4adcb-c8ea-4583-9473-cb91e59c5627-host-slash\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.145353 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145370 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145407 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-slash\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.145434 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:32:47.645412944 +0000 UTC m=+2.993056101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145526 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-slash\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145571 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-device-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145602 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03881413-57f5-4fc2-bc7d-94ae5db6c711-hosts-file\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145630 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qqh\" (UniqueName: \"kubernetes.io/projected/03881413-57f5-4fc2-bc7d-94ae5db6c711-kube-api-access-k4qqh\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145661 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7db4adcb-c8ea-4583-9473-cb91e59c5627-iptables-alerter-script\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145659 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-var-lib-kubelet\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145693 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145708 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-log-socket\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145762 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-modprobe-d\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.145971 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef405f5e-43cb-441b-9f77-109cacbde554-kubelet-dir\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146016 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-host\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146041 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-cnibin\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.148691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146123 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdcf203-d48e-4833-8762-4ef8e039151f-ovnkube-config\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146152 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-sysctl-conf\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146186 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdcf203-d48e-4833-8762-4ef8e039151f-node-log\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146340 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-os-release\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146479 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03881413-57f5-4fc2-bc7d-94ae5db6c711-hosts-file\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146518 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7db4adcb-c8ea-4583-9473-cb91e59c5627-host-slash\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146562 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-system-cni-dir\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146607 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.149357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.146716 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03881413-57f5-4fc2-bc7d-94ae5db6c711-tmp-dir\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.150669 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.150645 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdcf203-d48e-4833-8762-4ef8e039151f-ovn-node-metrics-cert\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.151587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.151546 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-tmp\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.152540 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.152520 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05aac2b6-8cf9-415c-9394-e8e39a0b0948-etc-tuned\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.165067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.165012 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbqk\" (UniqueName: \"kubernetes.io/projected/7db4adcb-c8ea-4583-9473-cb91e59c5627-kube-api-access-4pbqk\") pod \"iptables-alerter-d2jk7\" (UID: \"7db4adcb-c8ea-4583-9473-cb91e59c5627\") " pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.172013 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.171981 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82jw\" (UniqueName: \"kubernetes.io/projected/c3dac29b-b55f-40dd-8cc3-da48456c3b7a-kube-api-access-d82jw\") pod \"multus-additional-cni-plugins-zjlqm\" (UID: \"c3dac29b-b55f-40dd-8cc3-da48456c3b7a\") " pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.173114 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.173092 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qqh\" (UniqueName: \"kubernetes.io/projected/03881413-57f5-4fc2-bc7d-94ae5db6c711-kube-api-access-k4qqh\") pod \"node-resolver-qqkrj\" (UID: \"03881413-57f5-4fc2-bc7d-94ae5db6c711\") " pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.173204 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.173095 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzm4\" (UniqueName: \"kubernetes.io/projected/05aac2b6-8cf9-415c-9394-e8e39a0b0948-kube-api-access-fkzm4\") pod \"tuned-xxzw9\" (UID: \"05aac2b6-8cf9-415c-9394-e8e39a0b0948\") " pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.173564 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.173546 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz786\" (UniqueName: \"kubernetes.io/projected/9bdcf203-d48e-4833-8762-4ef8e039151f-kube-api-access-vz786\") pod \"ovnkube-node-9nzxn\" (UID: \"9bdcf203-d48e-4833-8762-4ef8e039151f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.173630 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.173594 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvx97\" (UniqueName: \"kubernetes.io/projected/df924705-2c5d-4fcb-a73a-41ec5bbd47d2-kube-api-access-wvx97\") pod \"multus-9749d\" (UID: \"df924705-2c5d-4fcb-a73a-41ec5bbd47d2\") " pod="openshift-multus/multus-9749d" Jul 06 19:32:47.174021 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.173984 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgcw\" (UniqueName: \"kubernetes.io/projected/da0c2dfe-f30f-4cfe-825e-a580daea42ea-kube-api-access-nlgcw\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.174630 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.174593 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/ef405f5e-43cb-441b-9f77-109cacbde554-kube-api-access-qsnkb\") pod \"aws-ebs-csi-driver-node-brsmc\" (UID: \"ef405f5e-43cb-441b-9f77-109cacbde554\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.227382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.227314 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" event={"ID":"172e65fcdcf827162778b41b6338da26","Type":"ContainerStarted","Data":"7e7a02c6eb83bf826869a7613086f647672dc3e1d8dc59019d9571308908faad"} Jul 06 19:32:47.228583 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.228548 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" event={"ID":"a6c8ec2323b990d6957cedec6f5b7c41","Type":"ContainerStarted","Data":"fad26cd7f37ee96d6d3cd094ca35b86f79eb8c546b4c821fae9c33c5b525e613"} Jul 06 19:32:47.246067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246041 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56edb1f5-00f9-4330-b62e-de016f1387af-konnectivity-ca\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246073 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83256f13-fd24-4c93-938d-c5ee1e8ecc93-host\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246089 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-kubelet-config\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246107 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56edb1f5-00f9-4330-b62e-de016f1387af-agent-certs\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-dbus\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246159 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:47.246177 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246164 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83256f13-fd24-4c93-938d-c5ee1e8ecc93-host\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.246383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246197 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-kubelet-config\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.246383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246217 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83256f13-fd24-4c93-938d-c5ee1e8ecc93-serviceca\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.246383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246243 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhj7\" (UniqueName: \"kubernetes.io/projected/83256f13-fd24-4c93-938d-c5ee1e8ecc93-kube-api-access-zkhj7\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.246383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246286 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.246500 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.246385 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:47.246500 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.246447 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:47.746425929 +0000 UTC m=+3.094069087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:47.246584 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246496 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6987347-09bc-4125-9ebc-00e37b431786-dbus\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.246641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246591 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/56edb1f5-00f9-4330-b62e-de016f1387af-konnectivity-ca\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.246860 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.246842 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83256f13-fd24-4c93-938d-c5ee1e8ecc93-serviceca\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.248596 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.248575 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/56edb1f5-00f9-4330-b62e-de016f1387af-agent-certs\") pod \"konnectivity-agent-6z7rf\" (UID: \"56edb1f5-00f9-4330-b62e-de016f1387af\") " pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.255340 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.255296 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:32:47.255340 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.255314 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:32:47.255340 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.255328 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:47.255485 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.255371 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:32:47.755359257 +0000 UTC m=+3.103002410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:47.261253 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.261235 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhj7\" (UniqueName: \"kubernetes.io/projected/83256f13-fd24-4c93-938d-c5ee1e8ecc93-kube-api-access-zkhj7\") pod \"node-ca-r2rtw\" (UID: \"83256f13-fd24-4c93-938d-c5ee1e8ecc93\") " pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.326856 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.326824 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" Jul 06 19:32:47.334982 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.334960 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" Jul 06 19:32:47.337850 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.337824 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dac29b_b55f_40dd_8cc3_da48456c3b7a.slice/crio-48448bae3a37c90fd028313a9b0acbac859bb62932ff8b3a13321524385832c2 WatchSource:0}: Error finding container 48448bae3a37c90fd028313a9b0acbac859bb62932ff8b3a13321524385832c2: Status 404 returned error can't find the container with id 48448bae3a37c90fd028313a9b0acbac859bb62932ff8b3a13321524385832c2 Jul 06 19:32:47.345473 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.345451 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef405f5e_43cb_441b_9f77_109cacbde554.slice/crio-0103fd5fee4cd56e5d653046f46f32f3fe17ab10027996ff24979ab50f3caf99 WatchSource:0}: Error finding container 0103fd5fee4cd56e5d653046f46f32f3fe17ab10027996ff24979ab50f3caf99: Status 404 returned error can't find the container with id 0103fd5fee4cd56e5d653046f46f32f3fe17ab10027996ff24979ab50f3caf99 Jul 06 19:32:47.347286 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.347267 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" Jul 06 19:32:47.354491 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.354467 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:32:47.358214 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.358192 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aac2b6_8cf9_415c_9394_e8e39a0b0948.slice/crio-d19ded2fcca3fd7cb47f84fb66639da0e6c59dd57154c35fba4bd661232196f8 WatchSource:0}: Error finding container d19ded2fcca3fd7cb47f84fb66639da0e6c59dd57154c35fba4bd661232196f8: Status 404 returned error can't find the container with id d19ded2fcca3fd7cb47f84fb66639da0e6c59dd57154c35fba4bd661232196f8 Jul 06 19:32:47.361986 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.361945 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9749d" Jul 06 19:32:47.364322 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.364296 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdcf203_d48e_4833_8762_4ef8e039151f.slice/crio-91d691900710f0553e9f29bf6f7cd4c17894843a9a5763d6a2bceda8b0bd8c37 WatchSource:0}: Error finding container 91d691900710f0553e9f29bf6f7cd4c17894843a9a5763d6a2bceda8b0bd8c37: Status 404 returned error can't find the container with id 91d691900710f0553e9f29bf6f7cd4c17894843a9a5763d6a2bceda8b0bd8c37 Jul 06 19:32:47.368865 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.368825 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qqkrj" Jul 06 19:32:47.372793 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.372770 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf924705_2c5d_4fcb_a73a_41ec5bbd47d2.slice/crio-89fc01300fc61508c40073333555b86abf4610bc2aa1a9162774ff8edd335b51 WatchSource:0}: Error finding container 89fc01300fc61508c40073333555b86abf4610bc2aa1a9162774ff8edd335b51: Status 404 returned error can't find the container with id 89fc01300fc61508c40073333555b86abf4610bc2aa1a9162774ff8edd335b51 Jul 06 19:32:47.379295 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.379271 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03881413_57f5_4fc2_bc7d_94ae5db6c711.slice/crio-85db4fd9550b393c9fd1d312a86d10d4bc0e15ea4e3f62fbdab7f7fd3ec7fe9a WatchSource:0}: Error finding container 85db4fd9550b393c9fd1d312a86d10d4bc0e15ea4e3f62fbdab7f7fd3ec7fe9a: Status 404 returned error can't find the container with id 85db4fd9550b393c9fd1d312a86d10d4bc0e15ea4e3f62fbdab7f7fd3ec7fe9a Jul 06 19:32:47.381859 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.381846 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d2jk7" Jul 06 19:32:47.387804 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.387783 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:32:47.393092 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.393062 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db4adcb_c8ea_4583_9473_cb91e59c5627.slice/crio-73c7233ab6052eec8ff3073ef7238fb9588e6e36d921b88b739e2faa25259aea WatchSource:0}: Error finding container 73c7233ab6052eec8ff3073ef7238fb9588e6e36d921b88b739e2faa25259aea: Status 404 returned error can't find the container with id 73c7233ab6052eec8ff3073ef7238fb9588e6e36d921b88b739e2faa25259aea Jul 06 19:32:47.394129 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.394065 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2rtw" Jul 06 19:32:47.401192 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.401163 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56edb1f5_00f9_4330_b62e_de016f1387af.slice/crio-a62cd7a5b65862cfc742d7a402d01e6a1400e05b811d56f60c3851bb941d1528 WatchSource:0}: Error finding container a62cd7a5b65862cfc742d7a402d01e6a1400e05b811d56f60c3851bb941d1528: Status 404 returned error can't find the container with id a62cd7a5b65862cfc742d7a402d01e6a1400e05b811d56f60c3851bb941d1528 Jul 06 19:32:47.406063 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:32:47.406040 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83256f13_fd24_4c93_938d_c5ee1e8ecc93.slice/crio-37a6c948315ea954d78b83e7631d595b4cedeaebbbe68801847094ee74dc2809 WatchSource:0}: Error finding container 37a6c948315ea954d78b83e7631d595b4cedeaebbbe68801847094ee74dc2809: Status 404 returned error can't find the container with id 37a6c948315ea954d78b83e7631d595b4cedeaebbbe68801847094ee74dc2809 Jul 06 19:32:47.559728 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.559650 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:32:47.648774 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.648741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:47.648967 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.648911 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:47.649038 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.648994 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.648974496 +0000 UTC m=+3.996617650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:47.749543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.749511 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:47.749712 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.749633 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:47.749712 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.749691 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.749675501 +0000 UTC m=+4.097318664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:47.850374 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:47.850276 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:47.850546 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.850453 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:32:47.850546 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.850478 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:32:47.850546 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.850491 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:47.850714 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:47.850555 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.850536255 +0000 UTC m=+4.198179411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:48.100130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.100074 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:27:47 +0000 UTC" deadline="2028-02-08 00:52:00.646643387 +0000 UTC" Jul 06 19:32:48.100130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.100114 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13949h19m12.546534042s" Jul 06 19:32:48.217757 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.217672 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:48.217757 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.217720 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:48.217986 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.217789 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:48.217986 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.217937 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:48.246888 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.246847 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" event={"ID":"ef405f5e-43cb-441b-9f77-109cacbde554","Type":"ContainerStarted","Data":"0103fd5fee4cd56e5d653046f46f32f3fe17ab10027996ff24979ab50f3caf99"} Jul 06 19:32:48.248357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.248325 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2rtw" event={"ID":"83256f13-fd24-4c93-938d-c5ee1e8ecc93","Type":"ContainerStarted","Data":"37a6c948315ea954d78b83e7631d595b4cedeaebbbe68801847094ee74dc2809"} Jul 06 19:32:48.249589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.249561 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z7rf" event={"ID":"56edb1f5-00f9-4330-b62e-de016f1387af","Type":"ContainerStarted","Data":"a62cd7a5b65862cfc742d7a402d01e6a1400e05b811d56f60c3851bb941d1528"} Jul 06 19:32:48.250642 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.250610 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qqkrj" event={"ID":"03881413-57f5-4fc2-bc7d-94ae5db6c711","Type":"ContainerStarted","Data":"85db4fd9550b393c9fd1d312a86d10d4bc0e15ea4e3f62fbdab7f7fd3ec7fe9a"} Jul 06 19:32:48.251702 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.251679 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9749d" event={"ID":"df924705-2c5d-4fcb-a73a-41ec5bbd47d2","Type":"ContainerStarted","Data":"89fc01300fc61508c40073333555b86abf4610bc2aa1a9162774ff8edd335b51"} Jul 06 19:32:48.252771 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.252748 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"91d691900710f0553e9f29bf6f7cd4c17894843a9a5763d6a2bceda8b0bd8c37"} Jul 06 19:32:48.253817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.253794 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" event={"ID":"05aac2b6-8cf9-415c-9394-e8e39a0b0948","Type":"ContainerStarted","Data":"d19ded2fcca3fd7cb47f84fb66639da0e6c59dd57154c35fba4bd661232196f8"} Jul 06 19:32:48.254890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.254864 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerStarted","Data":"48448bae3a37c90fd028313a9b0acbac859bb62932ff8b3a13321524385832c2"} Jul 06 19:32:48.255861 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.255842 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d2jk7" event={"ID":"7db4adcb-c8ea-4583-9473-cb91e59c5627","Type":"ContainerStarted","Data":"73c7233ab6052eec8ff3073ef7238fb9588e6e36d921b88b739e2faa25259aea"} Jul 06 19:32:48.656155 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.656069 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:48.656320 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.656243 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:48.656320 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.656304 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:32:50.656286923 +0000 UTC m=+6.003930079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:48.756741 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.756704 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:48.756950 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.756851 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:48.756950 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.756912 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:50.756895178 +0000 UTC m=+6.104538353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:48.859450 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:48.859352 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:48.859628 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.859536 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:32:48.859628 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.859556 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:32:48.859628 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.859569 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:48.859791 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:48.859631 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:32:50.85961253 +0000 UTC m=+6.207255708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:49.101368 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:49.101279 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:27:47 +0000 UTC" deadline="2028-01-19 18:41:56.632319122 +0000 UTC" Jul 06 19:32:49.101368 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:49.101339 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h9m7.530984054s" Jul 06 19:32:49.220112 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:49.220075 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:49.220281 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:49.220217 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:50.217779 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.217527 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:50.218289 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.217915 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:50.218362 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.218307 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:50.218461 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.218398 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:50.295666 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.295190 2584 generic.go:356] "Generic (PLEG): container finished" podID="172e65fcdcf827162778b41b6338da26" containerID="dcd3442edfb3ca90c9d8d2214a57fb18e19b906e570ce874fd060d37216eb0fd" exitCode=0 Jul 06 19:32:50.295666 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.295290 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" event={"ID":"172e65fcdcf827162778b41b6338da26","Type":"ContainerDied","Data":"dcd3442edfb3ca90c9d8d2214a57fb18e19b906e570ce874fd060d37216eb0fd"} Jul 06 19:32:50.302611 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.302586 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" event={"ID":"a6c8ec2323b990d6957cedec6f5b7c41","Type":"ContainerStarted","Data":"0823bece29462395082226e36301976e48dab4fdbaa0b88f3b9c388e09374303"} Jul 06 19:32:50.332044 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.331776 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-184.ec2.internal" podStartSLOduration=4.331755538 podStartE2EDuration="4.331755538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:50.330784705 +0000 UTC m=+5.678427880" watchObservedRunningTime="2026-07-06 19:32:50.331755538 +0000 UTC m=+5.679398714" Jul 06 19:32:50.677856 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.677773 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:50.678040 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.677988 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:50.678096 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.678050 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:32:54.678031601 +0000 UTC m=+10.025674762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:50.778462 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.778424 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:50.778631 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.778566 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:50.778631 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.778625 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:54.778607418 +0000 UTC m=+10.126250574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:50.879814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:50.879151 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:50.879814 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.879376 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:32:50.879814 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.879399 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:32:50.879814 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.879412 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:50.879814 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:50.879469 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:32:54.879451352 +0000 UTC m=+10.227094509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:51.217288 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:51.217250 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:51.217524 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:51.217478 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:51.311125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:51.311085 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" event={"ID":"172e65fcdcf827162778b41b6338da26","Type":"ContainerStarted","Data":"f96c22690e0dd2d45b1873083aa4eef466de86d8554526ec5ee489f378960939"} Jul 06 19:32:51.335497 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:51.335364 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-184.ec2.internal" podStartSLOduration=5.335344119 podStartE2EDuration="5.335344119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:51.335002712 +0000 UTC m=+6.682645890" watchObservedRunningTime="2026-07-06 19:32:51.335344119 +0000 UTC m=+6.682987294" Jul 06 19:32:52.217976 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:52.217943 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:52.217976 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:52.217966 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:52.218218 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:52.218076 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:52.218218 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:52.218192 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:53.217241 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:53.217209 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:53.217703 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:53.217343 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:54.218055 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:54.218018 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:54.218625 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.218153 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:54.218625 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:54.218207 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:54.218625 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.218309 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:54.715548 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:54.715462 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:54.715704 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.715634 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:54.715773 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.715712 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:33:02.715688692 +0000 UTC m=+18.063331845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:32:54.816686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:54.816582 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:54.816865 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.816697 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:54.816865 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.816777 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:02.816755958 +0000 UTC m=+18.164399114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:32:54.918027 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:54.917985 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:54.918358 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.918194 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:32:54.918358 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.918225 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:32:54.918358 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.918238 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:54.918358 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:54.918297 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:33:02.918279024 +0000 UTC m=+18.265922189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:32:55.217600 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:55.217565 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:55.221318 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:55.221158 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:56.217841 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:56.217592 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:56.217841 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:56.217642 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:56.217841 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:56.217730 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:56.218151 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:56.217859 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:57.217391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:57.217360 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:57.217818 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:57.217489 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:32:58.217669 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:58.217639 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:32:58.218142 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:58.217639 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:32:58.218142 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:58.217763 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:32:58.218142 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:58.217827 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:32:59.217221 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:32:59.217122 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:32:59.217370 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:32:59.217288 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:00.217277 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:00.217236 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:00.217727 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:00.217246 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:00.217727 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:00.217386 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:00.217727 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:00.217445 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:01.217583 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:01.217548 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:01.218080 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:01.217677 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:02.217230 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:02.217190 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:02.217457 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:02.217192 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:02.217457 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.217337 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:02.217457 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.217439 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:02.780762 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:02.780731 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:02.781249 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.780843 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:33:02.781249 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.780892 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:33:18.780878602 +0000 UTC m=+34.128521754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:33:02.881090 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:02.881056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:02.881257 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.881233 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:33:02.881323 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.881309 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:18.881288573 +0000 UTC m=+34.228931726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:33:02.981686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:02.981651 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:02.981893 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.981775 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:33:02.981893 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.981795 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:33:02.981893 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.981805 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:33:02.981893 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:02.981851 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:33:18.981836825 +0000 UTC m=+34.329479978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:33:03.217723 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:03.217632 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:03.217905 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:03.217782 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:04.217945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:04.217894 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:04.217945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:04.217913 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:04.218467 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:04.218038 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:04.218467 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:04.218188 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:05.218556 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:05.218511 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:05.219043 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:05.218598 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:06.217275 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:06.217236 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:06.217463 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:06.217236 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:06.217463 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:06.217381 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:06.217463 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:06.217450 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:07.217026 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:07.216996 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:07.217404 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:07.217121 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:07.342963 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:07.342674 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9749d" event={"ID":"df924705-2c5d-4fcb-a73a-41ec5bbd47d2","Type":"ContainerStarted","Data":"cab544f786bdd13558903acf53cb16162df809c388885e0c57d788623ee23ce7"} Jul 06 19:33:07.344985 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:07.344957 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"2dbc47a01ba5b3cf943e003cd009065740468fb4fada66d1aa8890d5d2de69be"} Jul 06 19:33:07.346685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:07.346656 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" event={"ID":"05aac2b6-8cf9-415c-9394-e8e39a0b0948","Type":"ContainerStarted","Data":"fc8087fab4343c13a9771c516311256899c8d8350bb20dde1b3cb4a627e403bb"} Jul 06 19:33:07.404286 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:07.403296 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9749d" podStartSLOduration=2.5863399400000002 podStartE2EDuration="22.403272108s" totalImagesPullingTime="19.816932168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:07.380694835 +0000 UTC m=+22.728338010" watchObservedRunningTime="2026-07-06 19:33:07.403272108 +0000 UTC m=+22.750915284" Jul 06 19:33:08.217840 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.217661 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:08.218488 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.217660 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:08.218488 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:08.217913 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:08.218488 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:08.218011 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:08.349409 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.349375 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z7rf" event={"ID":"56edb1f5-00f9-4330-b62e-de016f1387af","Type":"ContainerStarted","Data":"0b6e8a53671ba0abaf25f705ab39d7b516181cebfc0e923cfe552ed6e5d091e1"} Jul 06 19:33:08.350711 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.350689 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qqkrj" event={"ID":"03881413-57f5-4fc2-bc7d-94ae5db6c711","Type":"ContainerStarted","Data":"297a0d675923494c9b5c54768fe92024b4e928dc69afc5b7c96549789cb87983"} Jul 06 19:33:08.353639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.353615 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"27da4d97b7145059ecd62b8856445452d1c864cabb53659cbddd9547f8c02aee"} Jul 06 19:33:08.353737 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.353670 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"1739c260da8ea32dc6e76f87302f0276eb4063c5a5a2dff0cca347e27dff786c"} Jul 06 19:33:08.353737 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.353709 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"3abb6ceab43e9c58376455657dd0dd32d052e887a532c441a25d23e937d03ffb"} Jul 06 19:33:08.353817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.353779 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"650aa349dbc5d624f30ad21e1a1e53033a9ae2b91bbf3a9748745de1bf968f1e"} Jul 06 19:33:08.353817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.353814 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"6334ba249aa4569377b9e8389c155d88988bbb14b0b9471b028388fb59cbf93b"} Jul 06 19:33:08.354957 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.354917 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="dba8fb2bcd32dcf6d2f68f8852f5562b237c3fdf81487934b1fd2e94f7779cfb" exitCode=0 Jul 06 19:33:08.355037 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.354965 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"dba8fb2bcd32dcf6d2f68f8852f5562b237c3fdf81487934b1fd2e94f7779cfb"} Jul 06 19:33:08.356392 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.356366 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" event={"ID":"ef405f5e-43cb-441b-9f77-109cacbde554","Type":"ContainerStarted","Data":"79aa37ca5d3974914aa0450546a03fc8bfc19ac355e6d4f6074ac25fcd4c23d4"} Jul 06 19:33:08.357630 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.357609 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2rtw" event={"ID":"83256f13-fd24-4c93-938d-c5ee1e8ecc93","Type":"ContainerStarted","Data":"0f3e607ee2d39bbe9e037d415ba862bf96cd8e99dcfe1c7bff37cae4b6a60713"} Jul 06 19:33:08.366242 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.366201 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6z7rf" podStartSLOduration=3.624815298 podStartE2EDuration="23.366183424s" totalImagesPullingTime="19.741368126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:08.366006471 +0000 UTC m=+23.713649647" watchObservedRunningTime="2026-07-06 19:33:08.366183424 +0000 UTC m=+23.713826600" Jul 06 19:33:08.366398 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.366371 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xxzw9" podStartSLOduration=3.5835027200000003 podStartE2EDuration="23.36636299s" totalImagesPullingTime="19.78286027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:07.403991012 +0000 UTC m=+22.751634191" watchObservedRunningTime="2026-07-06 19:33:08.36636299 +0000 UTC m=+23.714006166" Jul 06 19:33:08.418848 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.418791 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qqkrj" podStartSLOduration=3.650146824 podStartE2EDuration="23.418770352s" totalImagesPullingTime="19.768623528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:08.400525237 +0000 UTC m=+23.748168412" watchObservedRunningTime="2026-07-06 19:33:08.418770352 +0000 UTC m=+23.766413528" Jul 06 19:33:08.419016 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.418879 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r2rtw" podStartSLOduration=3.6835620220000003 podStartE2EDuration="23.418874227s" totalImagesPullingTime="19.735312205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:08.417754537 +0000 UTC m=+23.765397723" watchObservedRunningTime="2026-07-06 19:33:08.418874227 +0000 UTC m=+23.766517403" Jul 06 19:33:08.654407 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:08.654379 2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:33:09.105004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.104891 2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:33:08.654403751Z","UUID":"ed34cd58-68a0-4cb5-ba55-aa02cd6de1bc","Handler":null,"Name":"","Endpoint":""} Jul 06 19:33:09.106515 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.106492 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:33:09.106616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.106534 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:33:09.217816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.217778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:09.218006 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:09.217908 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:09.362378 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.362286 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d2jk7" event={"ID":"7db4adcb-c8ea-4583-9473-cb91e59c5627","Type":"ContainerStarted","Data":"063e3e19b0eb6e204251bfdb95c66c42bb343acf091929047699d77d867ebed5"} Jul 06 19:33:09.364559 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.364531 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" event={"ID":"ef405f5e-43cb-441b-9f77-109cacbde554","Type":"ContainerStarted","Data":"bdc0902251136176a7e068fa2992440ff6c7ebdd2984f949cfeab377fc33b0a2"} Jul 06 19:33:09.529874 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.529833 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:09.692554 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:09.692508 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:10.217976 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.217888 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:10.218171 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:10.218003 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:10.218171 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.218035 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:10.218171 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:10.218129 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:10.369430 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.369392 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"11545704568fa951cc44515d0a7f35cf17c2175a1c2c1bea196f5336bb58e769"} Jul 06 19:33:10.371636 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.371599 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" event={"ID":"ef405f5e-43cb-441b-9f77-109cacbde554","Type":"ContainerStarted","Data":"0d5aab527dbb46280f2bbde0809774c496220603fabe2b37abd42a76a6cdc713"} Jul 06 19:33:10.391227 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.391170 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-brsmc" podStartSLOduration=3.371106868 podStartE2EDuration="25.391153136s" totalImagesPullingTime="22.020046268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:10.391070061 +0000 UTC m=+25.738713237" watchObservedRunningTime="2026-07-06 19:33:10.391153136 +0000 UTC m=+25.738796313" Jul 06 19:33:10.391676 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:10.391647 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d2jk7" podStartSLOduration=5.645005032 podStartE2EDuration="25.391637479s" totalImagesPullingTime="19.746632447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:09.379412875 +0000 UTC m=+24.727056054" watchObservedRunningTime="2026-07-06 19:33:10.391637479 +0000 UTC m=+25.739280708" Jul 06 19:33:11.217209 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:11.217178 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:11.217373 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:11.217296 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:12.217948 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.217564 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:12.217948 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.217564 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:12.217948 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:12.217915 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:12.219530 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:12.217963 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:12.254243 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.254194 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.324101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.324005 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.380732 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.380701 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" event={"ID":"9bdcf203-d48e-4833-8762-4ef8e039151f","Type":"ContainerStarted","Data":"1ce23e19532106f86331588cfd09930e1a3761f7f80565ca0928fcaf5b4edb3a"} Jul 06 19:33:12.381044 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.381024 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:12.397197 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.397174 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:12.408441 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.408401 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" podStartSLOduration=7.546227831 podStartE2EDuration="27.408382042s" totalImagesPullingTime="19.862154211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:12.408054393 +0000 UTC m=+27.755697569" watchObservedRunningTime="2026-07-06 19:33:12.408382042 +0000 UTC m=+27.756025238" Jul 06 19:33:12.414266 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.414231 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.477586 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.477548 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.933657 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.933611 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:33:12.934445 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.934426 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:33:12.985208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:12.985171 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:13.217603 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.217523 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:13.217862 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:13.217692 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:13.383836 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.383805 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:33:13.384413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.383862 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:13.384413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.383920 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:13.384413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.384016 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6z7rf" Jul 06 19:33:13.417002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.416972 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:13.452643 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:13.452609 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.218236 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.217988 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:14.218413 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:14.218367 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:14.218413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.218031 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:14.218554 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:14.218460 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:14.387311 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.387277 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="03a88dfd8c60d1d6425a3f9f525a319e797736338e9aeb0ef1ff5976babbe98e" exitCode=0 Jul 06 19:33:14.387625 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.387357 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"03a88dfd8c60d1d6425a3f9f525a319e797736338e9aeb0ef1ff5976babbe98e"} Jul 06 19:33:14.571152 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.571081 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpfsm"] Jul 06 19:33:14.571287 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.571172 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:14.571287 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:14.571257 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:14.574123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.574084 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zgx"] Jul 06 19:33:14.574234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.574192 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:14.574311 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:14.574291 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:14.577089 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.577044 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vmzt"] Jul 06 19:33:14.577216 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.577129 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:14.577275 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:14.577239 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:14.769111 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.769082 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.840337 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.840263 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.910892 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:14.910841 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:15.392144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:15.392055 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="ad17eae6556f8a2935c691dbc01c42358d1b5c29f895db8178ec56f95e03bbb9" exitCode=0 Jul 06 19:33:15.392144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:15.392125 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"ad17eae6556f8a2935c691dbc01c42358d1b5c29f895db8178ec56f95e03bbb9"} Jul 06 19:33:15.681986 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:15.681950 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:15.791970 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:15.791920 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:16.217191 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.217085 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:16.217191 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.217119 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:16.217191 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.217119 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:16.217400 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:16.217207 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:16.217400 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:16.217325 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:16.217497 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:16.217405 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:16.312947 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.312904 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:16.396985 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.396894 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="69e19ef714d8d733b7c8dc344a4c84fe94480454ca534a41fd2124771434f3a8" exitCode=0 Jul 06 19:33:16.396985 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:16.396957 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"69e19ef714d8d733b7c8dc344a4c84fe94480454ca534a41fd2124771434f3a8"} Jul 06 19:33:18.217617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.217427 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:18.218036 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.217427 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:18.218036 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.217703 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zgx" podUID="a6987347-09bc-4125-9ebc-00e37b431786" Jul 06 19:33:18.218036 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.217432 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:18.218036 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.217774 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpfsm" podUID="8b74d603-d4a3-49e9-8910-d97715b572ac" Jul 06 19:33:18.218036 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.217867 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vmzt" podUID="da0c2dfe-f30f-4cfe-825e-a580daea42ea" Jul 06 19:33:18.658802 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.658765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:18.717215 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.717146 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:18.794553 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.794465 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:18.794692 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.794609 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:33:18.794692 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.794669 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs podName:da0c2dfe-f30f-4cfe-825e-a580daea42ea nodeName:}" failed. No retries permitted until 2026-07-06 19:33:50.794652399 +0000 UTC m=+66.142295572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs") pod "network-metrics-daemon-9vmzt" (UID: "da0c2dfe-f30f-4cfe-825e-a580daea42ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:33:18.895228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.895183 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:18.895406 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.895359 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:33:18.895461 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.895440 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret podName:a6987347-09bc-4125-9ebc-00e37b431786 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:50.895418148 +0000 UTC m=+66.243061309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret") pod "global-pull-secret-syncer-89zgx" (UID: "a6987347-09bc-4125-9ebc-00e37b431786") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:33:18.996325 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:18.996290 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:18.996472 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.996454 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:33:18.996531 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.996477 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:33:18.996531 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.996490 2584 projected.go:196] Error preparing data for projected volume kube-api-access-b886h for pod openshift-network-diagnostics/network-check-target-wpfsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:33:18.996598 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:18.996539 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h podName:8b74d603-d4a3-49e9-8910-d97715b572ac nodeName:}" failed. No retries permitted until 2026-07-06 19:33:50.996524898 +0000 UTC m=+66.344168051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b886h" (UniqueName: "kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h") pod "network-check-target-wpfsm" (UID: "8b74d603-d4a3-49e9-8910-d97715b572ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:33:19.793063 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.793016 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:19.843043 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.843016 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-184.ec2.internal" event="NodeReady" Jul 06 19:33:19.843199 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.843171 2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:33:19.897388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.897304 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:19.899383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.899362 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h9jm9"] Jul 06 19:33:19.903529 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.903512 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:19.904185 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.904064 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xzt8q"] Jul 06 19:33:19.905455 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.905436 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f2dnr\"" Jul 06 19:33:19.905841 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.905825 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:33:19.905914 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.905830 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:33:19.908012 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.907991 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:19.910011 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.909991 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:33:19.910099 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.910072 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:33:19.910152 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.910108 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:33:19.910251 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.910224 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kpbzx\"" Jul 06 19:33:19.914297 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.914037 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9jm9"] Jul 06 19:33:19.917299 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:19.917277 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzt8q"] Jul 06 19:33:20.005998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.005963 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.005998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.005998 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxzrr\" (UniqueName: \"kubernetes.io/projected/35a4fb1f-a335-4e23-9018-1fd5d559645a-kube-api-access-zxzrr\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.006168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.006064 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a4fb1f-a335-4e23-9018-1fd5d559645a-config-volume\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.006168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.006083 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-cert\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.006168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.006105 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a4fb1f-a335-4e23-9018-1fd5d559645a-tmp-dir\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.006168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.006156 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvk4\" (UniqueName: \"kubernetes.io/projected/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-kube-api-access-gcvk4\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.107019 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.106981 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.107178 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107026 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxzrr\" (UniqueName: \"kubernetes.io/projected/35a4fb1f-a335-4e23-9018-1fd5d559645a-kube-api-access-zxzrr\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.107178 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:20.107136 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:33:20.107308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107192 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a4fb1f-a335-4e23-9018-1fd5d559645a-config-volume\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.107308 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:20.107202 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls podName:35a4fb1f-a335-4e23-9018-1fd5d559645a nodeName:}" failed. No retries permitted until 2026-07-06 19:33:20.607185057 +0000 UTC m=+35.954828216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls") pod "dns-default-h9jm9" (UID: "35a4fb1f-a335-4e23-9018-1fd5d559645a") : secret "dns-default-metrics-tls" not found Jul 06 19:33:20.107308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107242 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-cert\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.107308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107273 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a4fb1f-a335-4e23-9018-1fd5d559645a-tmp-dir\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.107500 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107406 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvk4\" (UniqueName: \"kubernetes.io/projected/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-kube-api-access-gcvk4\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.107635 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107619 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a4fb1f-a335-4e23-9018-1fd5d559645a-tmp-dir\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.107955 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.107913 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/35a4fb1f-a335-4e23-9018-1fd5d559645a-config-volume\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.112089 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.112065 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-cert\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.115480 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.115453 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxzrr\" (UniqueName: \"kubernetes.io/projected/35a4fb1f-a335-4e23-9018-1fd5d559645a-kube-api-access-zxzrr\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.115631 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.115608 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvk4\" (UniqueName: \"kubernetes.io/projected/1bb27ca0-ec9c-486a-8e30-e6d07fac3183-kube-api-access-gcvk4\") pod \"ingress-canary-xzt8q\" (UID: \"1bb27ca0-ec9c-486a-8e30-e6d07fac3183\") " pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.218127 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.218089 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:20.218127 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.218123 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:20.218319 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.218300 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:20.223518 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223367 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:33:20.223518 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223375 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:33:20.223518 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223403 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:33:20.223518 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223474 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:33:20.223823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223603 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-77m84\"" Jul 06 19:33:20.223823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.223676 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-965h5\"" Jul 06 19:33:20.247643 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.247619 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzt8q" Jul 06 19:33:20.410532 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.410505 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzt8q"] Jul 06 19:33:20.414803 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:20.414771 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb27ca0_ec9c_486a_8e30_e6d07fac3183.slice/crio-5b856956a0df63fef95501f88cce91b80900e0bc9123bf64cf22377c0ece1589 WatchSource:0}: Error finding container 5b856956a0df63fef95501f88cce91b80900e0bc9123bf64cf22377c0ece1589: Status 404 returned error can't find the container with id 5b856956a0df63fef95501f88cce91b80900e0bc9123bf64cf22377c0ece1589 Jul 06 19:33:20.611527 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:20.611435 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:20.611690 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:20.611614 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:33:20.611751 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:20.611693 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls podName:35a4fb1f-a335-4e23-9018-1fd5d559645a nodeName:}" failed. No retries permitted until 2026-07-06 19:33:21.611671983 +0000 UTC m=+36.959315139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls") pod "dns-default-h9jm9" (UID: "35a4fb1f-a335-4e23-9018-1fd5d559645a") : secret "dns-default-metrics-tls" not found Jul 06 19:33:21.337461 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.337263 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.339718 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.339693 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw"] Jul 06 19:33:21.374413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.374380 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw"] Jul 06 19:33:21.374582 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.374498 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" Jul 06 19:33:21.376801 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.376777 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 19:33:21.376950 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.376823 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 19:33:21.377371 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.377355 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9rwqg\"" Jul 06 19:33:21.410010 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.409970 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzt8q" event={"ID":"1bb27ca0-ec9c-486a-8e30-e6d07fac3183","Type":"ContainerStarted","Data":"5b856956a0df63fef95501f88cce91b80900e0bc9123bf64cf22377c0ece1589"} Jul 06 19:33:21.414319 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.414284 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.513500 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.513457 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.519421 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.519383 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9px\" (UniqueName: \"kubernetes.io/projected/1d0e3640-ba9d-4a1b-8655-770da0692518-kube-api-access-4h9px\") pod \"migrator-d57d58cc9-hjwmw\" (UID: \"1d0e3640-ba9d-4a1b-8655-770da0692518\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" Jul 06 19:33:21.571382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.571345 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zbfds"] Jul 06 19:33:21.589418 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.589266 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbfds"] Jul 06 19:33:21.589418 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.589412 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.591326 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.591300 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:33:21.591473 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.591301 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:33:21.591727 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.591707 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:33:21.591817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.591759 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:33:21.591817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.591801 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jsgwd\"" Jul 06 19:33:21.602367 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.602319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.620155 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.620124 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:21.620319 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.620224 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9px\" (UniqueName: \"kubernetes.io/projected/1d0e3640-ba9d-4a1b-8655-770da0692518-kube-api-access-4h9px\") pod \"migrator-d57d58cc9-hjwmw\" (UID: \"1d0e3640-ba9d-4a1b-8655-770da0692518\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" Jul 06 19:33:21.620319 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.620292 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:33:21.620425 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.620380 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls podName:35a4fb1f-a335-4e23-9018-1fd5d559645a nodeName:}" failed. No retries permitted until 2026-07-06 19:33:23.620358186 +0000 UTC m=+38.968001350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls") pod "dns-default-h9jm9" (UID: "35a4fb1f-a335-4e23-9018-1fd5d559645a") : secret "dns-default-metrics-tls" not found Jul 06 19:33:21.630307 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.630282 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9px\" (UniqueName: \"kubernetes.io/projected/1d0e3640-ba9d-4a1b-8655-770da0692518-kube-api-access-4h9px\") pod \"migrator-d57d58cc9-hjwmw\" (UID: \"1d0e3640-ba9d-4a1b-8655-770da0692518\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" Jul 06 19:33:21.685327 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.685295 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" Jul 06 19:33:21.721158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.721124 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-crio-socket\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.721294 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.721197 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.721294 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.721243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.721294 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.721277 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-data-volume\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.721413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.721311 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmzz\" (UniqueName: \"kubernetes.io/projected/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-api-access-nqmzz\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.821976 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.821908 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-data-volume\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.821993 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmzz\" (UniqueName: \"kubernetes.io/projected/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-api-access-nqmzz\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.822035 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-crio-socket\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.822094 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.822125 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822357 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.822198 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:33:21.822357 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.822260 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:22.322240905 +0000 UTC m=+37.669884074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : configmap "kube-rbac-proxy" not found Jul 06 19:33:21.822357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.822283 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-data-volume\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.822503 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.822405 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:33:21.822503 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:21.822456 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:22.322438381 +0000 UTC m=+37.670081537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : secret "insights-runtime-extractor-tls" not found Jul 06 19:33:21.822570 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.822531 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-crio-socket\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:21.833284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:21.833259 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmzz\" (UniqueName: \"kubernetes.io/projected/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-api-access-nqmzz\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:22.101896 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.101869 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw"] Jul 06 19:33:22.104908 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:22.104878 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e3640_ba9d_4a1b_8655_770da0692518.slice/crio-8cb37206fc4c24afa59869c2a7d17aa288d5f998b754199a1e37a943afa20adb WatchSource:0}: Error finding container 8cb37206fc4c24afa59869c2a7d17aa288d5f998b754199a1e37a943afa20adb: Status 404 returned error can't find the container with id 8cb37206fc4c24afa59869c2a7d17aa288d5f998b754199a1e37a943afa20adb Jul 06 19:33:22.178056 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.178022 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:22.270366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.270292 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:22.326041 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.326000 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:22.326041 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.326043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:22.326262 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:22.326180 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:33:22.326262 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:22.326254 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:23.326233979 +0000 UTC m=+38.673877135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : secret "insights-runtime-extractor-tls" not found Jul 06 19:33:22.326377 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:22.326182 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:33:22.326377 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:22.326330 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:23.326306961 +0000 UTC m=+38.673950115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : configmap "kube-rbac-proxy" not found Jul 06 19:33:22.414687 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.414639 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" event={"ID":"1d0e3640-ba9d-4a1b-8655-770da0692518","Type":"ContainerStarted","Data":"8cb37206fc4c24afa59869c2a7d17aa288d5f998b754199a1e37a943afa20adb"} Jul 06 19:33:22.917066 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:22.916995 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:23.012793 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.012755 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:23.331905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.331817 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:23.332090 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.331946 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:33:23.332090 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.331977 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:23.332090 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.332014 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:25.331994849 +0000 UTC m=+40.679638007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : configmap "kube-rbac-proxy" not found Jul 06 19:33:23.332248 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.332132 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:33:23.332248 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.332202 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:25.332178981 +0000 UTC m=+40.679822135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : secret "insights-runtime-extractor-tls" not found Jul 06 19:33:23.419356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.419325 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="ead4363008932dcb27a5691d3bfc4c97b69b16536a8981d779fc49478eb1df0b" exitCode=0 Jul 06 19:33:23.419864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.419366 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"ead4363008932dcb27a5691d3bfc4c97b69b16536a8981d779fc49478eb1df0b"} Jul 06 19:33:23.634560 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:23.634459 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:23.634719 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.634623 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:33:23.634719 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:23.634690 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls podName:35a4fb1f-a335-4e23-9018-1fd5d559645a nodeName:}" failed. No retries permitted until 2026-07-06 19:33:27.634671875 +0000 UTC m=+42.982315029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls") pod "dns-default-h9jm9" (UID: "35a4fb1f-a335-4e23-9018-1fd5d559645a") : secret "dns-default-metrics-tls" not found Jul 06 19:33:24.245454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.245374 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:24.423361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.423184 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzt8q" event={"ID":"1bb27ca0-ec9c-486a-8e30-e6d07fac3183","Type":"ContainerStarted","Data":"20273d544997e80227b06ed1d80191571a6e5d619191f58da64a3aa54c7978c6"} Jul 06 19:33:24.428298 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.428278 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" event={"ID":"1d0e3640-ba9d-4a1b-8655-770da0692518","Type":"ContainerStarted","Data":"7cb88ea524fd57782b59762f47aae0012d9e2f1fa29b7c06334f4294371dadcd"} Jul 06 19:33:24.432446 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.432377 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerStarted","Data":"b74a8b408ee3f2f158b06f1c25a90279669028f4af29cbb3d317ecb84ba5e16d"} Jul 06 19:33:24.464104 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.464077 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:24.545266 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.545231 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:24.566196 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.566172 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qqkrj_03881413-57f5-4fc2-bc7d-94ae5db6c711/dns-node-resolver/0.log" Jul 06 19:33:24.602222 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:24.602188 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:25.350080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.350042 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:25.350080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.350084 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:25.350282 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:25.350208 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:33:25.350282 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:25.350278 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:29.350260761 +0000 UTC m=+44.697903932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : secret "insights-runtime-extractor-tls" not found Jul 06 19:33:25.350574 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.350558 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:25.437723 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.437684 2584 generic.go:356] "Generic (PLEG): container finished" podID="c3dac29b-b55f-40dd-8cc3-da48456c3b7a" containerID="b74a8b408ee3f2f158b06f1c25a90279669028f4af29cbb3d317ecb84ba5e16d" exitCode=0 Jul 06 19:33:25.438459 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.437767 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerDied","Data":"b74a8b408ee3f2f158b06f1c25a90279669028f4af29cbb3d317ecb84ba5e16d"} Jul 06 19:33:25.439736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.439701 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" event={"ID":"1d0e3640-ba9d-4a1b-8655-770da0692518","Type":"ContainerStarted","Data":"5b9e3b4a519284e04714b15a4d1024d9503b4afc022a506117220c39be739b52"} Jul 06 19:33:25.485704 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.485654 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xzt8q" podStartSLOduration=2.578114077 podStartE2EDuration="6.485633532s" totalImagesPullingTime="3.907519455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:25.484964798 +0000 UTC m=+40.832607975" watchObservedRunningTime="2026-07-06 19:33:25.485633532 +0000 UTC m=+40.833276709" Jul 06 19:33:25.504391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.504339 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-hjwmw" podStartSLOduration=2.292501667 podStartE2EDuration="4.504319622s" totalImagesPullingTime="2.211817955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:25.50325625 +0000 UTC m=+40.850899425" watchObservedRunningTime="2026-07-06 19:33:25.504319622 +0000 UTC m=+40.851962797" Jul 06 19:33:25.566947 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.566910 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r2rtw_83256f13-fd24-4c93-938d-c5ee1e8ecc93/node-ca/0.log" Jul 06 19:33:25.602547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.602473 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:25.703146 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:25.703113 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:26.367037 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:26.367007 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzt8q_1bb27ca0-ec9c-486a-8e30-e6d07fac3183/serve-healthcheck-canary/0.log" Jul 06 19:33:26.444616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:26.444581 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" event={"ID":"c3dac29b-b55f-40dd-8cc3-da48456c3b7a","Type":"ContainerStarted","Data":"e520d28c60f8cbf7285e422274abbb7689491b51ee6ac960a4a816b78b0f04bd"} Jul 06 19:33:26.469800 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:26.469754 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjlqm" podStartSLOduration=8.477127107 podStartE2EDuration="41.469735869s" totalImagesPullingTime="32.992608762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:26.468254487 +0000 UTC m=+41.815897663" watchObservedRunningTime="2026-07-06 19:33:26.469735869 +0000 UTC m=+41.817379045" Jul 06 19:33:26.566453 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:26.566425 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-hjwmw_1d0e3640-ba9d-4a1b-8655-770da0692518/migrator/0.log" Jul 06 19:33:26.766087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:26.766060 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-hjwmw_1d0e3640-ba9d-4a1b-8655-770da0692518/graceful-termination/0.log" Jul 06 19:33:27.668324 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:27.668283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:27.668734 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:27.668429 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:33:27.668734 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:27.668530 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls podName:35a4fb1f-a335-4e23-9018-1fd5d559645a nodeName:}" failed. No retries permitted until 2026-07-06 19:33:35.668507733 +0000 UTC m=+51.016150888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls") pod "dns-default-h9jm9" (UID: "35a4fb1f-a335-4e23-9018-1fd5d559645a") : secret "dns-default-metrics-tls" not found Jul 06 19:33:29.005413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.005368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.188396 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.188368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.339152 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.339059 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.382135 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.382104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:29.382244 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:29.382226 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:33:29.382297 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:29.382288 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls podName:29ddcb7d-7155-4fd8-9ae3-8c55069f30c0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:37.382274093 +0000 UTC m=+52.729917251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbfds" (UID: "29ddcb7d-7155-4fd8-9ae3-8c55069f30c0") : secret "insights-runtime-extractor-tls" not found Jul 06 19:33:29.567210 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.567171 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.669411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.669374 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.832941 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:29.832899 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:30.310016 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:30.309978 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:30.904221 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:30.904187 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:31.309971 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:31.309921 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:31.707310 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:31.707273 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.114824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:32.114739 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.311317 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:32.311283 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.706253 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:32.706215 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.934802 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:32.934771 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:33.035169 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:33.035092 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:33.314772 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:33.314684 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:33.398156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:33.398128 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:33.652054 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:33.651969 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:33.757920 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:33.757889 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:34.114356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:34.114309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:34.305537 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:34.305504 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:34.699138 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:34.699103 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:34.947015 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:34.946976 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:35.317804 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.317773 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:35.733872 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.733837 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:35.737338 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.737308 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a4fb1f-a335-4e23-9018-1fd5d559645a-metrics-tls\") pod \"dns-default-h9jm9\" (UID: \"35a4fb1f-a335-4e23-9018-1fd5d559645a\") " pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:35.817942 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.817903 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:35.914782 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.914757 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:35.948440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:35.948412 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9jm9"] Jul 06 19:33:35.951824 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:35.951801 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a4fb1f_a335_4e23_9018_1fd5d559645a.slice/crio-9d933424052782a1fb79557f63b3079a9380f34e9fa4d8be6fa4cb00d2c26238 WatchSource:0}: Error finding container 9d933424052782a1fb79557f63b3079a9380f34e9fa4d8be6fa4cb00d2c26238: Status 404 returned error can't find the container with id 9d933424052782a1fb79557f63b3079a9380f34e9fa4d8be6fa4cb00d2c26238 Jul 06 19:33:36.121153 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:36.121066 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:36.303093 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:36.303064 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:36.469205 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:36.469163 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9jm9" event={"ID":"35a4fb1f-a335-4e23-9018-1fd5d559645a","Type":"ContainerStarted","Data":"9d933424052782a1fb79557f63b3079a9380f34e9fa4d8be6fa4cb00d2c26238"} Jul 06 19:33:36.500501 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:36.500475 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:36.642535 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:36.642465 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:37.309207 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:37.309167 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:37.447281 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:37.447249 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:37.449310 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:37.449282 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ddcb7d-7155-4fd8-9ae3-8c55069f30c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbfds\" (UID: \"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0\") " pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:37.501132 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:37.501104 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbfds" Jul 06 19:33:37.909647 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:37.909607 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:38.098406 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.098386 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbfds"] Jul 06 19:33:38.100758 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:38.100731 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ddcb7d_7155_4fd8_9ae3_8c55069f30c0.slice/crio-7fd210970ba8d72d581f4086f953073d88f3496275c4ff415b2152d068b4e079 WatchSource:0}: Error finding container 7fd210970ba8d72d581f4086f953073d88f3496275c4ff415b2152d068b4e079: Status 404 returned error can't find the container with id 7fd210970ba8d72d581f4086f953073d88f3496275c4ff415b2152d068b4e079 Jul 06 19:33:38.105159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.105133 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:38.303187 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.303101 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:38.475090 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.475053 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9jm9" event={"ID":"35a4fb1f-a335-4e23-9018-1fd5d559645a","Type":"ContainerStarted","Data":"894f2955abb868e11bc30351d7c733afcdaadc586869b110aa9127d0b8347684"} Jul 06 19:33:38.476288 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.476262 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbfds" event={"ID":"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0","Type":"ContainerStarted","Data":"7fd210970ba8d72d581f4086f953073d88f3496275c4ff415b2152d068b4e079"} Jul 06 19:33:38.514963 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:38.514918 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:39.348502 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.348458 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:39.480891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.480853 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9jm9" event={"ID":"35a4fb1f-a335-4e23-9018-1fd5d559645a","Type":"ContainerStarted","Data":"405bcfa8bb82e78f612588ebb4066932da77263b53fdfb940c8fb08b9c125e7e"} Jul 06 19:33:39.481166 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.481094 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:39.502414 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.502359 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h9jm9" podStartSLOduration=18.13073118 podStartE2EDuration="20.502340972s" totalImagesPullingTime="2.371609792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:39.50224338 +0000 UTC m=+54.849886570" watchObservedRunningTime="2026-07-06 19:33:39.502340972 +0000 UTC m=+54.849984147" Jul 06 19:33:39.507777 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.507748 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:39.708044 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:39.708004 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:42.491724 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:42.491692 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbfds" event={"ID":"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0","Type":"ContainerStarted","Data":"7b96344923b7189ca8407a345780cbba29b71cb0c2543c8ede7c6842a755a557"} Jul 06 19:33:42.492124 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:42.491745 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbfds" event={"ID":"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0","Type":"ContainerStarted","Data":"a5cb3bc138d8cb8aea23ecee3025f6311c57f212aef869f13c0f1e62cb62cc90"} Jul 06 19:33:45.405510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:45.405485 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9nzxn" Jul 06 19:33:45.509613 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:45.509565 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbfds" event={"ID":"29ddcb7d-7155-4fd8-9ae3-8c55069f30c0","Type":"ContainerStarted","Data":"23822d1905fd7c0cb96c7c86237d74c45ed81ecbee69cc473744072867f32bcb"} Jul 06 19:33:45.531875 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:45.531826 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zbfds" podStartSLOduration=17.433942144 podStartE2EDuration="24.531807976s" totalImagesPullingTime="7.097865832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:33:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:45.530963677 +0000 UTC m=+60.878606853" watchObservedRunningTime="2026-07-06 19:33:45.531807976 +0000 UTC m=+60.879451151" Jul 06 19:33:47.700417 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.700386 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v"] Jul 06 19:33:47.702399 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.702384 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.708952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.708463 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:33:47.708952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.708630 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:33:47.708952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.708746 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:33:47.708952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.708846 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:33:47.709188 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.709013 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:33:47.709188 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.709069 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:33:47.709556 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.709538 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:33:47.722532 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.722490 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v"] Jul 06 19:33:47.725492 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.725607 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725511 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.725607 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725589 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.725708 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725647 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.725708 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgjh\" (UniqueName: \"kubernetes.io/projected/e7446d93-f265-470b-a25b-a8e16e7ac0fe-kube-api-access-8kgjh\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.725797 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.725762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7446d93-f265-470b-a25b-a8e16e7ac0fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.826792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.826758 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.826792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.826794 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.827053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.826812 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.827053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.826907 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.827053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.826962 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgjh\" (UniqueName: \"kubernetes.io/projected/e7446d93-f265-470b-a25b-a8e16e7ac0fe-kube-api-access-8kgjh\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.827053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.827001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7446d93-f265-470b-a25b-a8e16e7ac0fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.827766 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.827731 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7446d93-f265-470b-a25b-a8e16e7ac0fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.829417 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.829389 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-ca\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.829565 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.829542 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.829696 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.829681 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.829796 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.829779 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7446d93-f265-470b-a25b-a8e16e7ac0fe-hub\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.833792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.833765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:47.838877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.838856 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgjh\" (UniqueName: \"kubernetes.io/projected/e7446d93-f265-470b-a25b-a8e16e7ac0fe-kube-api-access-8kgjh\") pod \"cluster-proxy-proxy-agent-75b55b64f7-brr7v\" (UID: \"e7446d93-f265-470b-a25b-a8e16e7ac0fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:47.929673 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:47.929636 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.019398 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.019320 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" Jul 06 19:33:48.026809 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.026776 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.131437 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.131398 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.141306 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.141278 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v"] Jul 06 19:33:48.145084 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:48.145058 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7446d93_f265_470b_a25b_a8e16e7ac0fe.slice/crio-c65a2ffbba65a9d645da67cd2351811db7020c667fa2ac3d6ae2c0201fd704c6 WatchSource:0}: Error finding container c65a2ffbba65a9d645da67cd2351811db7020c667fa2ac3d6ae2c0201fd704c6: Status 404 returned error can't find the container with id c65a2ffbba65a9d645da67cd2351811db7020c667fa2ac3d6ae2c0201fd704c6 Jul 06 19:33:48.231506 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.231472 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.520243 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.520210 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" event={"ID":"e7446d93-f265-470b-a25b-a8e16e7ac0fe","Type":"ContainerStarted","Data":"c65a2ffbba65a9d645da67cd2351811db7020c667fa2ac3d6ae2c0201fd704c6"} Jul 06 19:33:48.656438 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.656405 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.754417 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.754385 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.853347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.853190 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:48.936675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:48.936635 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.058920 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.058884 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.067597 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.067567 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj"] Jul 06 19:33:49.069602 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.069581 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:49.071772 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.071741 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:33:49.071910 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.071895 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5s996\"" Jul 06 19:33:49.081334 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.081307 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj"] Jul 06 19:33:49.136442 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.136349 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/408e8f38-c7b9-4ec4-af9a-45d0e4c1c919-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-lq5qj\" (UID: \"408e8f38-c7b9-4ec4-af9a-45d0e4c1c919\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:49.163638 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.163598 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.237457 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.237419 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/408e8f38-c7b9-4ec4-af9a-45d0e4c1c919-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-lq5qj\" (UID: \"408e8f38-c7b9-4ec4-af9a-45d0e4c1c919\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:49.240497 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.240466 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/408e8f38-c7b9-4ec4-af9a-45d0e4c1c919-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-lq5qj\" (UID: \"408e8f38-c7b9-4ec4-af9a-45d0e4c1c919\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:49.269718 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.269678 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.343577 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.343537 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.382959 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.382909 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:49.488079 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.488018 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h9jm9" Jul 06 19:33:49.524172 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.524127 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.549180 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.548978 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj"] Jul 06 19:33:49.554318 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:49.554285 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408e8f38_c7b9_4ec4_af9a_45d0e4c1c919.slice/crio-fb82fb436f861c60a6ddcb4e66009abf58c09f6002823a34a587e616c30d6855 WatchSource:0}: Error finding container fb82fb436f861c60a6ddcb4e66009abf58c09f6002823a34a587e616c30d6855: Status 404 returned error can't find the container with id fb82fb436f861c60a6ddcb4e66009abf58c09f6002823a34a587e616c30d6855 Jul 06 19:33:49.621504 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.621399 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.874278 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.874206 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:49.944957 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:49.944907 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.269572 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.269536 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.369663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.369628 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.485413 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.485380 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.532917 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.532823 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" event={"ID":"408e8f38-c7b9-4ec4-af9a-45d0e4c1c919","Type":"ContainerStarted","Data":"fb82fb436f861c60a6ddcb4e66009abf58c09f6002823a34a587e616c30d6855"} Jul 06 19:33:50.540084 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.540052 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.711467 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.711433 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.853331 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.853065 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:50.855274 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.855098 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:33:50.866150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.866087 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0c2dfe-f30f-4cfe-825e-a580daea42ea-metrics-certs\") pod \"network-metrics-daemon-9vmzt\" (UID: \"da0c2dfe-f30f-4cfe-825e-a580daea42ea\") " pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:50.896303 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.896270 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.954464 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.954426 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:50.956747 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.956728 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:33:50.968056 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.968036 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6987347-09bc-4125-9ebc-00e37b431786-original-pull-secret\") pod \"global-pull-secret-syncer-89zgx\" (UID: \"a6987347-09bc-4125-9ebc-00e37b431786\") " pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:50.997388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:50.997341 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:51.055810 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.055777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:51.058844 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.058822 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:33:51.068480 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.068453 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:33:51.080442 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.080412 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b886h\" (UniqueName: \"kubernetes.io/projected/8b74d603-d4a3-49e9-8910-d97715b572ac-kube-api-access-b886h\") pod \"network-check-target-wpfsm\" (UID: \"8b74d603-d4a3-49e9-8910-d97715b572ac\") " pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:51.097792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.097763 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:51.130135 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.130066 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-77m84\"" Jul 06 19:33:51.135361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.135328 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-965h5\"" Jul 06 19:33:51.137819 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.137798 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zgx" Jul 06 19:33:51.138878 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.138852 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:51.143798 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.143778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vmzt" Jul 06 19:33:51.312194 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.312162 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:51.374945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:51.374889 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:52.311698 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:52.311665 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:52.384230 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:52.384189 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:52.735264 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:52.735191 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vmzt"] Jul 06 19:33:52.739506 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:52.739481 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0c2dfe_f30f_4cfe_825e_a580daea42ea.slice/crio-3c89331fa3cce6d097fa58dc9ce50f8eed57127cdaa9facf931f1426d81730cb WatchSource:0}: Error finding container 3c89331fa3cce6d097fa58dc9ce50f8eed57127cdaa9facf931f1426d81730cb: Status 404 returned error can't find the container with id 3c89331fa3cce6d097fa58dc9ce50f8eed57127cdaa9facf931f1426d81730cb Jul 06 19:33:52.749175 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:52.749114 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zgx"] Jul 06 19:33:52.751034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:52.750953 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpfsm"] Jul 06 19:33:52.753063 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:52.753038 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6987347_09bc_4125_9ebc_00e37b431786.slice/crio-e708440ffbcde4dd1855db17edfa8f21f00ec3b7d7e6da8a2f17420160f50acc WatchSource:0}: Error finding container e708440ffbcde4dd1855db17edfa8f21f00ec3b7d7e6da8a2f17420160f50acc: Status 404 returned error can't find the container with id e708440ffbcde4dd1855db17edfa8f21f00ec3b7d7e6da8a2f17420160f50acc Jul 06 19:33:52.755351 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:52.755324 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b74d603_d4a3_49e9_8910_d97715b572ac.slice/crio-515b0b089283ede9eaae31d0068a766debf40860c13e1ff91429d3f6048a21d1 WatchSource:0}: Error finding container 515b0b089283ede9eaae31d0068a766debf40860c13e1ff91429d3f6048a21d1: Status 404 returned error can't find the container with id 515b0b089283ede9eaae31d0068a766debf40860c13e1ff91429d3f6048a21d1 Jul 06 19:33:53.076470 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.076381 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:53.160460 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.160417 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:53.557391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.557348 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" event={"ID":"e7446d93-f265-470b-a25b-a8e16e7ac0fe","Type":"ContainerStarted","Data":"f8096a112c61e79076451492b9f18090235080b7e791d6129d985a5d690e3746"} Jul 06 19:33:53.558001 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.557424 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" event={"ID":"e7446d93-f265-470b-a25b-a8e16e7ac0fe","Type":"ContainerStarted","Data":"acc011a03a4af0648031e0616480b5e4cb1a1dcd3287b7ac3639a7386761d4cf"} Jul 06 19:33:53.558001 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.557444 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" event={"ID":"e7446d93-f265-470b-a25b-a8e16e7ac0fe","Type":"ContainerStarted","Data":"40bc6d487b438ad15b18e53256a9c2092041bf76b5ff2d254f0fa71c2a5f6fe6"} Jul 06 19:33:53.560586 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.560556 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" event={"ID":"408e8f38-c7b9-4ec4-af9a-45d0e4c1c919","Type":"ContainerStarted","Data":"287bad9240b335b140f27e98bcffb728a151113a4afbe64fd7671d01dbf2f9e6"} Jul 06 19:33:53.560993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.560976 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:53.563754 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.563708 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zgx" event={"ID":"a6987347-09bc-4125-9ebc-00e37b431786","Type":"ContainerStarted","Data":"e708440ffbcde4dd1855db17edfa8f21f00ec3b7d7e6da8a2f17420160f50acc"} Jul 06 19:33:53.565720 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.565660 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vmzt" event={"ID":"da0c2dfe-f30f-4cfe-825e-a580daea42ea","Type":"ContainerStarted","Data":"3c89331fa3cce6d097fa58dc9ce50f8eed57127cdaa9facf931f1426d81730cb"} Jul 06 19:33:53.567282 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.567259 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" Jul 06 19:33:53.567746 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.567724 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpfsm" event={"ID":"8b74d603-d4a3-49e9-8910-d97715b572ac","Type":"ContainerStarted","Data":"515b0b089283ede9eaae31d0068a766debf40860c13e1ff91429d3f6048a21d1"} Jul 06 19:33:53.581412 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.581134 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75b55b64f7-brr7v" podStartSLOduration=2.168133363 podStartE2EDuration="6.581115294s" totalImagesPullingTime="4.412981931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:53.579947332 +0000 UTC m=+68.927590505" watchObservedRunningTime="2026-07-06 19:33:53.581115294 +0000 UTC m=+68.928758470" Jul 06 19:33:53.599609 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:53.599564 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-lq5qj" podStartSLOduration=1.595686963 podStartE2EDuration="4.599545438s" totalImagesPullingTime="3.003858475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:53.598792345 +0000 UTC m=+68.946435523" watchObservedRunningTime="2026-07-06 19:33:53.599545438 +0000 UTC m=+68.947188614" Jul 06 19:33:54.137708 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.137621 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:54.238755 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.238708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:54.339245 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.339186 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:54.574289 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.574250 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vmzt" event={"ID":"da0c2dfe-f30f-4cfe-825e-a580daea42ea","Type":"ContainerStarted","Data":"a7e88c7f846e8a8ad9edf4459c29924549de380081dea5c95e31854f62e7765f"} Jul 06 19:33:54.574753 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.574310 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vmzt" event={"ID":"da0c2dfe-f30f-4cfe-825e-a580daea42ea","Type":"ContainerStarted","Data":"85dfad7ba73894fe5e0c6c41ea93197e11f139f98203cf5b7ee7a690eb51c93d"} Jul 06 19:33:54.599345 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.599283 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9vmzt" podStartSLOduration=68.517205429 podStartE2EDuration="1m9.599261512s" totalImagesPullingTime="1.082056083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:54.598468918 +0000 UTC m=+69.946112093" watchObservedRunningTime="2026-07-06 19:33:54.599261512 +0000 UTC m=+69.946904689" Jul 06 19:33:54.763612 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.763572 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:54.847437 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:54.847347 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.590215 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:56.590172 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:57.591334 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:57.591229 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zgx" event={"ID":"a6987347-09bc-4125-9ebc-00e37b431786","Type":"ContainerStarted","Data":"23088b2843b525227cd47cf828590d94225352dbf870bba3e3907a499262a584"} Jul 06 19:33:57.592654 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:57.592633 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpfsm" event={"ID":"8b74d603-d4a3-49e9-8910-d97715b572ac","Type":"ContainerStarted","Data":"39676c453a8d23abbf20a5ef7208faf560a83bd42db5b03a50fdfcc3a236cc73"} Jul 06 19:33:57.592772 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:57.592752 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:33:57.614827 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:57.614776 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-89zgx" podStartSLOduration=68.080004385 podStartE2EDuration="1m12.614758975s" totalImagesPullingTime="4.53475459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:57.613825895 +0000 UTC m=+72.961469195" watchObservedRunningTime="2026-07-06 19:33:57.614758975 +0000 UTC m=+72.962402151" Jul 06 19:33:57.636948 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:57.636877 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wpfsm" podStartSLOduration=68.108578197 podStartE2EDuration="1m12.63686223s" totalImagesPullingTime="4.528284033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:57.636458497 +0000 UTC m=+72.984101672" watchObservedRunningTime="2026-07-06 19:33:57.63686223 +0000 UTC m=+72.984505405" Jul 06 19:33:58.047334 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.047294 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.178156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.178122 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.324185 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.324102 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.593826 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.593744 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8jwnb"] Jul 06 19:33:58.599057 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.599034 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.601440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.601412 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:33:58.601440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.601425 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:33:58.601590 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.601430 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:33:58.601590 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.601435 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:33:58.602015 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.601996 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:33:58.602116 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.602023 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:33:58.602116 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.602036 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qrzm9\"" Jul 06 19:33:58.607321 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.607297 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.683126 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.683098 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.714248 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-textfile\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714248 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714245 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714412 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714303 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714412 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714390 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-root\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714416 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-sys\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714447 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-metrics-client-ca\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714490 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-wtmp\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.714657 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.714536 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crzn\" (UniqueName: \"kubernetes.io/projected/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-kube-api-access-7crzn\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.784623 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.784592 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:58.815193 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crzn\" (UniqueName: \"kubernetes.io/projected/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-kube-api-access-7crzn\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815216 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-textfile\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815247 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815311 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-root\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815336 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-sys\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-sys\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815398 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-root\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:58.815404 2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815431 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-metrics-client-ca\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:58.815492 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls podName:e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:59.315467161 +0000 UTC m=+74.663110323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls") pod "node-exporter-8jwnb" (UID: "e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3") : secret "node-exporter-tls" not found Jul 06 19:33:58.815579 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815538 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-wtmp\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815656 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-textfile\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.815818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815700 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-wtmp\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.816010 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.815993 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.816087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.816070 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-metrics-client-ca\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.817646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.817631 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.832396 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.832371 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crzn\" (UniqueName: \"kubernetes.io/projected/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-kube-api-access-7crzn\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:58.885023 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:58.884911 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.006574 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.006535 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.079905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.079865 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.161130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.161101 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.305324 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.305289 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.319271 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.319248 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:59.321463 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.321444 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3-node-exporter-tls\") pod \"node-exporter-8jwnb\" (UID: \"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3\") " pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:59.508632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.508539 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8jwnb" Jul 06 19:33:59.519186 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:33:59.519160 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91e8398_7794_4b60_8a0c_d1fdcc7fa9a3.slice/crio-303d6ec17798a0d9789962b2dbd655c14337b9c6b9d4c0797992af4e3cf7fcd6 WatchSource:0}: Error finding container 303d6ec17798a0d9789962b2dbd655c14337b9c6b9d4c0797992af4e3cf7fcd6: Status 404 returned error can't find the container with id 303d6ec17798a0d9789962b2dbd655c14337b9c6b9d4c0797992af4e3cf7fcd6 Jul 06 19:33:59.582456 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.582420 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.600296 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.600263 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jwnb" event={"ID":"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3","Type":"ContainerStarted","Data":"303d6ec17798a0d9789962b2dbd655c14337b9c6b9d4c0797992af4e3cf7fcd6"} Jul 06 19:33:59.678347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.678318 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.689076 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.689043 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:33:59.691783 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.691768 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695646 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695654 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695672 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695682 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695723 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695728 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:33:59.695790 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695751 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vltn5\"" Jul 06 19:33:59.696239 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.695939 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:33:59.696239 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.696162 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:33:59.696239 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.696162 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:33:59.707913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.707882 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:33:59.722601 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722576 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722613 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722638 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722733 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722769 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722800 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722832 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722857 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.722898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722894 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.723101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722913 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.723101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.723101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722972 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzkh\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.723101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.722988 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.751972 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.751944 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:59.823752 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823675 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823752 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823718 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823754 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823783 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823821 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823846 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823868 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823897 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.823980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.824336 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.823998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.824336 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.824026 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.824336 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.824051 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzkh\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.824509 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.824427 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.824509 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:33:59.824443 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle podName:7c04830e-4543-49a7-b7e3-9f23c8e9060a nodeName:}" failed. No retries permitted until 2026-07-06 19:34:00.324421516 +0000 UTC m=+75.672064673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a") : configmap references non-existent config key: ca-bundle.crt Jul 06 19:33:59.824638 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.824603 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827078 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827173 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827379 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827494 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827475 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827611 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827586 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.827668 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.827653 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.828053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.828026 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.828138 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.828089 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.828208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.828193 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:33:59.832421 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:33:59.832403 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzkh\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:34:00.328625 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:00.328585 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:34:00.329438 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:00.329415 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:34:00.600817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:00.600715 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:34:00.787716 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:00.787684 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:34:00.790200 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:34:00.790174 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c04830e_4543_49a7_b7e3_9f23c8e9060a.slice/crio-2529ea6811c16a6d1e92b49e8478ed5ce924a72de6ac7cd54e2d07327a348ea1 WatchSource:0}: Error finding container 2529ea6811c16a6d1e92b49e8478ed5ce924a72de6ac7cd54e2d07327a348ea1: Status 404 returned error can't find the container with id 2529ea6811c16a6d1e92b49e8478ed5ce924a72de6ac7cd54e2d07327a348ea1 Jul 06 19:34:01.545132 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.545101 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:01.611233 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.611193 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"2529ea6811c16a6d1e92b49e8478ed5ce924a72de6ac7cd54e2d07327a348ea1"} Jul 06 19:34:01.613252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.613225 2584 generic.go:356] "Generic (PLEG): container finished" podID="e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3" containerID="6901ad50441a4577890da6a010b31715f1e531e0637e8c13d286d79bea056010" exitCode=0 Jul 06 19:34:01.613386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.613329 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jwnb" event={"ID":"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3","Type":"ContainerDied","Data":"6901ad50441a4577890da6a010b31715f1e531e0637e8c13d286d79bea056010"} Jul 06 19:34:01.634897 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.634864 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:01.698034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.698001 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:01.809963 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:01.809856 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:02.618881 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:02.618849 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5" exitCode=0 Jul 06 19:34:02.619374 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:02.618955 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5"} Jul 06 19:34:02.621321 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:02.621297 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jwnb" event={"ID":"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3","Type":"ContainerStarted","Data":"48065dc01e38fde5e7f340133ce385e4511861ca7af9825642b7048424262758"} Jul 06 19:34:02.621386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:02.621344 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8jwnb" event={"ID":"e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3","Type":"ContainerStarted","Data":"ed2f3465d65fd6593a789404d6b8c67f6e8204fd54b78d83292a20ad648c2088"} Jul 06 19:34:02.668743 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:02.668597 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8jwnb" podStartSLOduration=3.433643137 podStartE2EDuration="4.66857367s" totalImagesPullingTime="1.169168731s" totalInitContainerRuntime="65.761802ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:02.668459979 +0000 UTC m=+78.016103170" watchObservedRunningTime="2026-07-06 19:34:02.66857367 +0000 UTC m=+78.016216843" Jul 06 19:34:03.088094 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.088060 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b8479df95-n4w9q"] Jul 06 19:34:03.091095 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.091078 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.094216 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094192 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:34:03.094364 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094232 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a7o85cth1vpok\"" Jul 06 19:34:03.094364 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094254 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:34:03.094364 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094256 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:34:03.094364 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094192 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qkfz6\"" Jul 06 19:34:03.094364 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.094190 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:34:03.105901 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.105874 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b8479df95-n4w9q"] Jul 06 19:34:03.151452 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzvc\" (UniqueName: \"kubernetes.io/projected/bb0b0248-7619-4236-a16d-573fad578c5b-kube-api-access-7zzvc\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151452 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151454 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-client-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151486 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-tls\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151528 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-client-certs\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151556 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb0b0248-7619-4236-a16d-573fad578c5b-audit-log\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151624 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-metrics-server-audit-profiles\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.151686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.151675 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-client-certs\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252395 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252227 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb0b0248-7619-4236-a16d-573fad578c5b-audit-log\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252395 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252268 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-metrics-server-audit-profiles\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252395 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252310 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252534 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252453 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzvc\" (UniqueName: \"kubernetes.io/projected/bb0b0248-7619-4236-a16d-573fad578c5b-kube-api-access-7zzvc\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252534 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252499 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-client-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252626 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252537 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-tls\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.252823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.252796 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb0b0248-7619-4236-a16d-573fad578c5b-audit-log\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.253270 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.253237 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.253605 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.253562 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb0b0248-7619-4236-a16d-573fad578c5b-metrics-server-audit-profiles\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.255225 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.255200 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-client-certs\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.255331 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.255244 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-client-ca-bundle\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.255331 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.255282 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb0b0248-7619-4236-a16d-573fad578c5b-secret-metrics-server-tls\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.272147 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.272121 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzvc\" (UniqueName: \"kubernetes.io/projected/bb0b0248-7619-4236-a16d-573fad578c5b-kube-api-access-7zzvc\") pod \"metrics-server-7b8479df95-n4w9q\" (UID: \"bb0b0248-7619-4236-a16d-573fad578c5b\") " pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.373034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.372908 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:03.400105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.400079 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:03.444203 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.444167 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:03.540230 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.540199 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b8479df95-n4w9q"] Jul 06 19:34:03.542677 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:34:03.542648 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0b0248_7619_4236_a16d_573fad578c5b.slice/crio-1b9ee8f1485edc7c6db685b007995d6ca366bbb147c3192a32b634d84824a284 WatchSource:0}: Error finding container 1b9ee8f1485edc7c6db685b007995d6ca366bbb147c3192a32b634d84824a284: Status 404 returned error can't find the container with id 1b9ee8f1485edc7c6db685b007995d6ca366bbb147c3192a32b634d84824a284 Jul 06 19:34:03.626514 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.626432 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" event={"ID":"bb0b0248-7619-4236-a16d-573fad578c5b","Type":"ContainerStarted","Data":"1b9ee8f1485edc7c6db685b007995d6ca366bbb147c3192a32b634d84824a284"} Jul 06 19:34:03.660984 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.660956 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:03.758877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:03.758847 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:04.031496 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.031452 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:04.098645 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.098605 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:04.382333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.382306 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:04.635123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.635073 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe"} Jul 06 19:34:04.635551 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.635148 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333"} Jul 06 19:34:04.635551 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.635188 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c"} Jul 06 19:34:04.635551 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.635217 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85"} Jul 06 19:34:04.635551 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.635240 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad"} Jul 06 19:34:04.872487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.872406 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:34:04.876262 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.876239 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.878157 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878133 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:34:04.878403 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878138 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:34:04.878403 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878140 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:34:04.878588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878511 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:34:04.878588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878545 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:34:04.878588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878553 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:34:04.878588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878553 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:34:04.878854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878598 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:34:04.878854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878555 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:34:04.879068 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.878921 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nnq86\"" Jul 06 19:34:04.879068 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.879003 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:34:04.879278 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.879130 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:34:04.879278 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.879186 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:34:04.879629 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.879613 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:34:04.879888 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.879872 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ba1todjcu3ft\"" Jul 06 19:34:04.881472 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.881453 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:34:04.894351 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.894310 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:34:04.918188 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.918152 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:04.967864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.967830 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtnf\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.967864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.967871 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.967917 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.967961 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.967989 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968022 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968110 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968151 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968214 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968259 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968295 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968317 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968352 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968391 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968377 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968396 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968429 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968458 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968472 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968521 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:04.968719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:04.968560 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.069854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.069816 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.069854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.069858 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.069920 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.069981 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070010 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070230 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070280 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070306 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070435 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070339 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070435 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070371 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070435 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070408 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtnf\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070439 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070515 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070580 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070614 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070655 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070687 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.070818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070726 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.071046 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.070874 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.073487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.073169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.073487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.073283 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.073487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.073328 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.073721 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.073666 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.074241 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.073942 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.074241 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.074144 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.074392 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.074321 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.074554 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.074529 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.075096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.074771 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.075096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.074789 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.075260 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.075231 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.075649 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.075415 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.075649 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.075494 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.076108 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.076079 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.076195 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.076153 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.077552 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.077520 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.077660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.077616 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.077660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.077646 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.079793 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.079770 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtnf\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf\") pod \"prometheus-k8s-0\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.186694 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.186664 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:05.369654 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.369623 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:34:05.371636 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:34:05.371601 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b98b92_6b84_4912_a3dd_1fff57bdb0d3.slice/crio-ed2c9a44dedb10b24946d455cc37f607e93beb451e4a72891a5787a51f0cdc8b WatchSource:0}: Error finding container ed2c9a44dedb10b24946d455cc37f607e93beb451e4a72891a5787a51f0cdc8b: Status 404 returned error can't find the container with id ed2c9a44dedb10b24946d455cc37f607e93beb451e4a72891a5787a51f0cdc8b Jul 06 19:34:05.641162 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.641126 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" exitCode=0 Jul 06 19:34:05.641597 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.641213 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} Jul 06 19:34:05.641597 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.641293 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"ed2c9a44dedb10b24946d455cc37f607e93beb451e4a72891a5787a51f0cdc8b"} Jul 06 19:34:05.642749 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.642732 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" event={"ID":"bb0b0248-7619-4236-a16d-573fad578c5b","Type":"ContainerStarted","Data":"afb9ab669734225bc47106fcab8a78994a67aef25a8f3b8afe06b576e7dbf8a8"} Jul 06 19:34:05.733079 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.733048 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:05.832104 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:05.832058 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:06.649896 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:06.649749 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerStarted","Data":"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf"} Jul 06 19:34:06.674903 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:06.674837 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" podStartSLOduration=1.935915456 podStartE2EDuration="3.674816077s" totalImagesPullingTime="1.738900621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:05.726146126 +0000 UTC m=+81.073789312" watchObservedRunningTime="2026-07-06 19:34:06.674816077 +0000 UTC m=+82.022459254" Jul 06 19:34:06.676525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:06.676460 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.713970635 podStartE2EDuration="7.676443453s" totalImagesPullingTime="3.947496876s" totalInitContainerRuntime="14.975942ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:06.673841145 +0000 UTC m=+82.021484320" watchObservedRunningTime="2026-07-06 19:34:06.676443453 +0000 UTC m=+82.024086631" Jul 06 19:34:09.333586 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:09.333490 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:09.398396 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:09.398304 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:09.666474 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:09.666446 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} Jul 06 19:34:09.666622 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:09.666506 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} Jul 06 19:34:10.068726 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:10.068642 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:10.167087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:10.167043 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:11.676351 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.676321 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} Jul 06 19:34:11.676832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.676395 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} Jul 06 19:34:11.676832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.676439 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} Jul 06 19:34:11.676832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.676482 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerStarted","Data":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} Jul 06 19:34:11.707497 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.707449 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.268443718 podStartE2EDuration="7.707428872s" totalImagesPullingTime="5.420171639s" totalInitContainerRuntime="18.813515ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:11.70506154 +0000 UTC m=+87.052704718" watchObservedRunningTime="2026-07-06 19:34:11.707428872 +0000 UTC m=+87.055072046" Jul 06 19:34:11.895395 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.895353 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:11.958990 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:11.958894 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:13.165904 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:13.165868 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:13.250295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:13.250263 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.187637 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:15.187590 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:34:15.576151 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:15.576074 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.674573 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:15.674538 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.024203 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:16.024165 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.125382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:16.125348 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.584657 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:17.584623 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.685648 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:17.685614 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:18.654651 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:18.654612 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:18.761125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:18.761089 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.272597 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:19.272566 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.374054 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:19.374024 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.677157 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:19.677126 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.746800 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:19.746769 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:20.387402 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:20.387368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.016542 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:21.016512 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.119105 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:21.119071 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:23.400519 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:23.400487 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:23.400886 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:23.400542 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:27.288722 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:27.288689 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzt8q_1bb27ca0-ec9c-486a-8e30-e6d07fac3183/serve-healthcheck-canary/0.log" Jul 06 19:34:28.598047 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:28.598019 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wpfsm" Jul 06 19:34:33.653629 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:33.653589 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:33.754306 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:33.754274 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:43.405908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:43.405881 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:43.409695 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:43.409674 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b8479df95-n4w9q" Jul 06 19:34:48.657272 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:48.657234 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:48.718385 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:48.718347 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:55.156563 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:55.156518 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:55.259297 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:55.259252 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:55.324344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:34:55.324319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:05.187697 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:05.187665 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:05.206646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:05.206614 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:05.897010 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:05.896981 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:17.912785 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:17.912746 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:18.017071 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:18.017038 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:19.040576 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040538 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:19.041087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040865 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="alertmanager" containerID="cri-o://644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" gracePeriod=120 Jul 06 19:35:19.041087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040903 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-web" containerID="cri-o://3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" gracePeriod=120 Jul 06 19:35:19.041087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040866 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="prom-label-proxy" containerID="cri-o://28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" gracePeriod=120 Jul 06 19:35:19.041087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040866 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy" containerID="cri-o://02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" gracePeriod=120 Jul 06 19:35:19.041087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.040906 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-metric" containerID="cri-o://ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" gracePeriod=120 Jul 06 19:35:19.041339 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.041028 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="config-reloader" containerID="cri-o://88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" gracePeriod=120 Jul 06 19:35:19.931828 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931798 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" exitCode=0 Jul 06 19:35:19.931828 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931821 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" exitCode=0 Jul 06 19:35:19.931828 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931828 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" exitCode=0 Jul 06 19:35:19.931828 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931834 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" exitCode=0 Jul 06 19:35:19.932159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931878 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf"} Jul 06 19:35:19.932159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.931981 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333"} Jul 06 19:35:19.932159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.932011 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85"} Jul 06 19:35:19.932159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:19.932036 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad"} Jul 06 19:35:20.290140 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.290111 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:20.379018 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.378985 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379018 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379020 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379043 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379074 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379105 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379139 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379164 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379206 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379243 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379297 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkzkh\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379336 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379363 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379390 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls\") pod \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\" (UID: \"7c04830e-4543-49a7-b7e3-9f23c8e9060a\") " Jul 06 19:35:20.379788 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379538 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:20.379788 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379647 2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.379975 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379949 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:20.379975 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.379961 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:20.381865 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.381815 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume" (OuterVolumeSpecName: "config-volume") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382819 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out" (OuterVolumeSpecName: "config-out") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382839 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382843 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh" (OuterVolumeSpecName: "kube-api-access-gkzkh") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "kube-api-access-gkzkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382867 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382885 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.382908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.382898 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:20.383452 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.383433 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.386841 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.386820 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.392834 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.392814 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config" (OuterVolumeSpecName: "web-config") pod "7c04830e-4543-49a7-b7e3-9f23c8e9060a" (UID: "7c04830e-4543-49a7-b7e3-9f23c8e9060a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480708 2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480741 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzkh\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-kube-api-access-gkzkh\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480752 2584 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c04830e-4543-49a7-b7e3-9f23c8e9060a-metrics-client-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480760 2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-alertmanager-main-db\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480769 2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-main-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480778 2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c04830e-4543-49a7-b7e3-9f23c8e9060a-tls-assets\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480786 2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-out\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.480792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480793 2584 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-config-volume\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.481167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480801 2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.481167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480811 2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.481167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480820 2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-web-config\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.481167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.480827 2584 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c04830e-4543-49a7-b7e3-9f23c8e9060a-cluster-tls-config\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:20.939066 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939032 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" exitCode=0 Jul 06 19:35:20.939066 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939059 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerID="3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" exitCode=0 Jul 06 19:35:20.939284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939106 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:20.939284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939117 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe"} Jul 06 19:35:20.939284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939202 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c"} Jul 06 19:35:20.939284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939233 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c04830e-4543-49a7-b7e3-9f23c8e9060a","Type":"ContainerDied","Data":"2529ea6811c16a6d1e92b49e8478ed5ce924a72de6ac7cd54e2d07327a348ea1"} Jul 06 19:35:20.939284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.939277 2584 scope.go:117] "RemoveContainer" containerID="28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" Jul 06 19:35:20.947205 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.947187 2584 scope.go:117] "RemoveContainer" containerID="ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" Jul 06 19:35:20.954057 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.954041 2584 scope.go:117] "RemoveContainer" containerID="02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" Jul 06 19:35:20.960614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.960597 2584 scope.go:117] "RemoveContainer" containerID="3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" Jul 06 19:35:20.963554 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.963518 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:20.969369 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.969339 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:20.972488 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.972469 2584 scope.go:117] "RemoveContainer" containerID="88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" Jul 06 19:35:20.979198 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.979176 2584 scope.go:117] "RemoveContainer" containerID="644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" Jul 06 19:35:20.985917 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.985901 2584 scope.go:117] "RemoveContainer" containerID="ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5" Jul 06 19:35:20.992457 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.992435 2584 scope.go:117] "RemoveContainer" containerID="28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" Jul 06 19:35:20.992826 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.992785 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf\": container with ID starting with 28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf not found: ID does not exist" containerID="28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" Jul 06 19:35:20.992905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.992834 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf"} err="failed to get container status \"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf\": rpc error: code = NotFound desc = could not find container \"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf\": container with ID starting with 28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf not found: ID does not exist" Jul 06 19:35:20.992905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.992897 2584 scope.go:117] "RemoveContainer" containerID="ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" Jul 06 19:35:20.993344 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.993317 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe\": container with ID starting with ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe not found: ID does not exist" containerID="ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" Jul 06 19:35:20.993425 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.993356 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe"} err="failed to get container status \"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe\": rpc error: code = NotFound desc = could not find container \"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe\": container with ID starting with ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe not found: ID does not exist" Jul 06 19:35:20.993425 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.993378 2584 scope.go:117] "RemoveContainer" containerID="02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" Jul 06 19:35:20.993730 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.993697 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333\": container with ID starting with 02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333 not found: ID does not exist" containerID="02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" Jul 06 19:35:20.993826 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.993730 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333"} err="failed to get container status \"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333\": rpc error: code = NotFound desc = could not find container \"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333\": container with ID starting with 02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333 not found: ID does not exist" Jul 06 19:35:20.993826 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.993752 2584 scope.go:117] "RemoveContainer" containerID="3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" Jul 06 19:35:20.994078 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.994059 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c\": container with ID starting with 3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c not found: ID does not exist" containerID="3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" Jul 06 19:35:20.994167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994082 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c"} err="failed to get container status \"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c\": rpc error: code = NotFound desc = could not find container \"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c\": container with ID starting with 3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c not found: ID does not exist" Jul 06 19:35:20.994167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994097 2584 scope.go:117] "RemoveContainer" containerID="88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" Jul 06 19:35:20.994350 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.994326 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85\": container with ID starting with 88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85 not found: ID does not exist" containerID="88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" Jul 06 19:35:20.994399 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994352 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85"} err="failed to get container status \"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85\": rpc error: code = NotFound desc = could not find container \"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85\": container with ID starting with 88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85 not found: ID does not exist" Jul 06 19:35:20.994399 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994364 2584 scope.go:117] "RemoveContainer" containerID="644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" Jul 06 19:35:20.994654 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.994628 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad\": container with ID starting with 644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad not found: ID does not exist" containerID="644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" Jul 06 19:35:20.994753 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994657 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad"} err="failed to get container status \"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad\": rpc error: code = NotFound desc = could not find container \"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad\": container with ID starting with 644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad not found: ID does not exist" Jul 06 19:35:20.994753 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994687 2584 scope.go:117] "RemoveContainer" containerID="ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5" Jul 06 19:35:20.994978 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994961 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:20.994978 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:20.994971 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5\": container with ID starting with ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5 not found: ID does not exist" containerID="ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5" Jul 06 19:35:20.995088 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.994991 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5"} err="failed to get container status \"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5\": rpc error: code = NotFound desc = could not find container \"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5\": container with ID starting with ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5 not found: ID does not exist" Jul 06 19:35:20.995088 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995005 2584 scope.go:117] "RemoveContainer" containerID="28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf" Jul 06 19:35:20.995265 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995250 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf"} err="failed to get container status \"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf\": rpc error: code = NotFound desc = could not find container \"28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf\": container with ID starting with 28a33b84ad04abaae3289279548452f841688f81c5a3a24931e156c114b65ebf not found: ID does not exist" Jul 06 19:35:20.995311 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995266 2584 scope.go:117] "RemoveContainer" containerID="ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe" Jul 06 19:35:20.995311 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995266 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="alertmanager" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995312 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="alertmanager" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995322 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-web" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995330 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-web" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995340 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="prom-label-proxy" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995349 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="prom-label-proxy" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995361 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995367 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy" Jul 06 19:35:20.995386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995382 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-metric" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995390 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-metric" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995405 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="init-config-reloader" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995411 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="init-config-reloader" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995422 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="config-reloader" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995429 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="config-reloader" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995499 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="alertmanager" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995495 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe"} err="failed to get container status \"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe\": rpc error: code = NotFound desc = could not find container \"ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe\": container with ID starting with ef833eda64ec202bc57463fbf839943752429adb67c2fd1e2b86535115d5c4fe not found: ID does not exist" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995514 2584 scope.go:117] "RemoveContainer" containerID="02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995506 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-web" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995562 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="config-reloader" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995570 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="prom-label-proxy" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995577 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy-metric" Jul 06 19:35:20.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995585 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" containerName="kube-rbac-proxy" Jul 06 19:35:20.996110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995748 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333"} err="failed to get container status \"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333\": rpc error: code = NotFound desc = could not find container \"02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333\": container with ID starting with 02826e074461f24b68dd7d2547b21eda7dbe8cb1b232bd6e246b002076ca7333 not found: ID does not exist" Jul 06 19:35:20.996110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.995769 2584 scope.go:117] "RemoveContainer" containerID="3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c" Jul 06 19:35:20.996110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996015 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c"} err="failed to get container status \"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c\": rpc error: code = NotFound desc = could not find container \"3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c\": container with ID starting with 3bfc4fcdc1ddfae3530832c60b40c076476d0e2352ff5e0b1b6505993db6bf3c not found: ID does not exist" Jul 06 19:35:20.996110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996030 2584 scope.go:117] "RemoveContainer" containerID="88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85" Jul 06 19:35:20.996298 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996271 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85"} err="failed to get container status \"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85\": rpc error: code = NotFound desc = could not find container \"88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85\": container with ID starting with 88530746e3f08a0c8febaf6b29c521f48b23e8cb01e10ad9c9bc0a19c0503f85 not found: ID does not exist" Jul 06 19:35:20.996362 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996301 2584 scope.go:117] "RemoveContainer" containerID="644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad" Jul 06 19:35:20.996522 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996505 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad"} err="failed to get container status \"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad\": rpc error: code = NotFound desc = could not find container \"644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad\": container with ID starting with 644a04b6769dc942d5501f057ea6d700b4ec11abf64e1158a2e13c55a99818ad not found: ID does not exist" Jul 06 19:35:20.996590 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996523 2584 scope.go:117] "RemoveContainer" containerID="ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5" Jul 06 19:35:20.996759 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.996738 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5"} err="failed to get container status \"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5\": rpc error: code = NotFound desc = could not find container \"ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5\": container with ID starting with ae3424323235abd0354d70159bdd6fd600e6c208fc0f332c5762747e2deb31c5 not found: ID does not exist" Jul 06 19:35:20.999058 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:20.999044 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.001327 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001306 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001333 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001346 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001359 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001423 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vltn5\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001312 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001366 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:35:21.001454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001362 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:35:21.001863 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.001776 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:35:21.007249 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.007231 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:35:21.013035 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.013004 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:21.084672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084636 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-web-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084679 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084776 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084807 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084843 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-out\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.084873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084870 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084890 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084920 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.084962 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.085001 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tml7\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-kube-api-access-9tml7\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.085067 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.085053 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186157 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186157 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186160 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186186 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186212 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186232 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tml7\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-kube-api-access-9tml7\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186257 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186407 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-web-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186465 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186498 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186539 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186564 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.186624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.186608 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-out\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.187449 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.187105 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189253 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189277 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-out\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189285 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189333 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189339 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189376 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189810 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189582 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189810 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189681 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-web-config\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189810 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189696 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.189920 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.189847 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.190095 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.190074 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebea2914-f9b7-478e-b0bc-a35512ff3a89-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.191289 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.191268 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebea2914-f9b7-478e-b0bc-a35512ff3a89-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.194557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.194537 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tml7\" (UniqueName: \"kubernetes.io/projected/ebea2914-f9b7-478e-b0bc-a35512ff3a89-kube-api-access-9tml7\") pod \"alertmanager-main-0\" (UID: \"ebea2914-f9b7-478e-b0bc-a35512ff3a89\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.222018 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.221993 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c04830e-4543-49a7-b7e3-9f23c8e9060a" path="/var/lib/kubelet/pods/7c04830e-4543-49a7-b7e3-9f23c8e9060a/volumes" Jul 06 19:35:21.308703 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.308674 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:35:21.444779 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.444688 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:35:21.448534 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:35:21.448506 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebea2914_f9b7_478e_b0bc_a35512ff3a89.slice/crio-c31305bb229ac60e27f8e1441e6724a4a0ad2db679d88852ad86a8fbf27d7713 WatchSource:0}: Error finding container c31305bb229ac60e27f8e1441e6724a4a0ad2db679d88852ad86a8fbf27d7713: Status 404 returned error can't find the container with id c31305bb229ac60e27f8e1441e6724a4a0ad2db679d88852ad86a8fbf27d7713 Jul 06 19:35:21.944505 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.944476 2584 generic.go:356] "Generic (PLEG): container finished" podID="ebea2914-f9b7-478e-b0bc-a35512ff3a89" containerID="002810440d84d9457aea581cf461bab3bbdec2712d57a21f98cdade5050f78ce" exitCode=0 Jul 06 19:35:21.944699 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.944573 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerDied","Data":"002810440d84d9457aea581cf461bab3bbdec2712d57a21f98cdade5050f78ce"} Jul 06 19:35:21.944699 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:21.944646 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"c31305bb229ac60e27f8e1441e6724a4a0ad2db679d88852ad86a8fbf27d7713"} Jul 06 19:35:22.952111 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952076 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"cb920421affd9ec52625cdd80119fa9b1c5c674ac8090714c018faa5253a5ac1"} Jul 06 19:35:22.952499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952144 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"b57cdac0c13cb92ee0927881f717c72b8a3dca2e409600a0512da2220ce0f455"} Jul 06 19:35:22.952499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952176 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"868bb5b2788ed0bfdd350aaf2cf3026c1df57bf9ada91df53e8e14473ed8e6ad"} Jul 06 19:35:22.952499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952203 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"a769d3522a74343f669b52784626e5bbf9659f11328cb099bb896bb0342d6654"} Jul 06 19:35:22.952499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"44b58afe02696df6db29a52568f3d387adef59e61b0fb7a948b89f47933d9976"} Jul 06 19:35:22.952499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.952259 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebea2914-f9b7-478e-b0bc-a35512ff3a89","Type":"ContainerStarted","Data":"7ced0d9d46c329c9c7f69afefdffebca04076b12b9299d20cfd9617f53b9cd14"} Jul 06 19:35:22.984121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:22.984068 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.968449075 podStartE2EDuration="2.98404937s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.600295ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:22.982135448 +0000 UTC m=+158.329778635" watchObservedRunningTime="2026-07-06 19:35:22.98404937 +0000 UTC m=+158.331692545" Jul 06 19:35:23.371888 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.371808 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:23.372175 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372140 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="prometheus" containerID="cri-o://8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" gracePeriod=600 Jul 06 19:35:23.372175 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372159 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" gracePeriod=600 Jul 06 19:35:23.372175 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372175 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="thanos-sidecar" containerID="cri-o://001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" gracePeriod=600 Jul 06 19:35:23.372514 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372196 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-web" containerID="cri-o://4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" gracePeriod=600 Jul 06 19:35:23.372514 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372200 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy" containerID="cri-o://6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" gracePeriod=600 Jul 06 19:35:23.372514 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.372176 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="config-reloader" containerID="cri-o://2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" gracePeriod=600 Jul 06 19:35:23.609649 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.609625 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:23.712687 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712641 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712687 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712694 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712737 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712753 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712770 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712785 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712807 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712828 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712878 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712965 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.712993 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.712998 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxtnf\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713050 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713060 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713087 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713115 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713141 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713182 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713210 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713248 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713277 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713299 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca\") pod \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\" (UID: \"12b98b92-6b84-4912-a3dd-1fff57bdb0d3\") " Jul 06 19:35:23.713912 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713469 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.713912 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713579 2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.713912 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713599 2584 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.713912 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.713829 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.714212 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.714187 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.715000 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.714275 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:23.715000 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.714286 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.715168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.715058 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.715649 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.715621 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:35:23.715732 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.715668 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.716734 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.716708 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:23.717003 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.716971 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.717113 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717059 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out" (OuterVolumeSpecName: "config-out") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:23.717113 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717074 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf" (OuterVolumeSpecName: "kube-api-access-vxtnf") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "kube-api-access-vxtnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:23.717256 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717145 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.717454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717434 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.717706 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717680 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.717870 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.717854 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.718338 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.718312 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config" (OuterVolumeSpecName: "config") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.718500 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.718475 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.725786 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.725764 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config" (OuterVolumeSpecName: "web-config") pod "12b98b92-6b84-4912-a3dd-1fff57bdb0d3" (UID: "12b98b92-6b84-4912-a3dd-1fff57bdb0d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:35:23.814333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814289 2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config-out\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814329 2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814339 2584 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-config\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814349 2584 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814358 2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-db\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814366 2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-web-config\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814375 2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814386 2584 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-kube-rbac-proxy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814395 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtnf\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-kube-api-access-vxtnf\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814404 2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814415 2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814422 2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-tls-assets\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814432 2584 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814442 2584 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814451 2584 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-grpc-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814459 2584 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814468 2584 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-secret-metrics-client-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.814593 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.814477 2584 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b98b92-6b84-4912-a3dd-1fff57bdb0d3-configmap-metrics-client-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:35:23.959137 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959108 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" exitCode=0 Jul 06 19:35:23.959137 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959133 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" exitCode=0 Jul 06 19:35:23.959137 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959139 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" exitCode=0 Jul 06 19:35:23.959137 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959147 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" exitCode=0 Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959155 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" exitCode=0 Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959161 2584 generic.go:356] "Generic (PLEG): container finished" podID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" exitCode=0 Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959179 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959192 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959262 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959290 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959314 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959328 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959364 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} Jul 06 19:35:23.959685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.959387 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"12b98b92-6b84-4912-a3dd-1fff57bdb0d3","Type":"ContainerDied","Data":"ed2c9a44dedb10b24946d455cc37f607e93beb451e4a72891a5787a51f0cdc8b"} Jul 06 19:35:23.967156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.967130 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:23.974432 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.974413 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:23.980936 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.980905 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:23.987323 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.987302 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:23.992100 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.992076 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:23.996873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.996852 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:23.997205 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:23.997177 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.005043 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.005016 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.011725 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.011707 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.012026 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.012007 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.012093 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012041 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.012093 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012067 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.012351 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.012335 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.012402 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012358 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.012402 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012375 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.012617 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.012596 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.012658 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012622 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.012658 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012640 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.012843 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.012829 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.012884 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012846 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.012884 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.012858 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.013103 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.013087 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.013151 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013108 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.013151 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013122 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.013361 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.013344 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.013411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013366 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.013411 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013382 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.013609 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:35:24.013591 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.013650 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013621 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.013650 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013636 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.013851 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013834 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.013891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.013851 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.014081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014062 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.014139 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014080 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.014263 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014248 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.014305 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014262 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.014466 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014449 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.014509 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014467 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.014686 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014668 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.014723 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014686 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.014868 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014853 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.014904 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.014868 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.015062 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015047 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.015113 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015062 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.015266 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015250 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.015315 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015265 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.015457 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015442 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.015495 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015458 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.015635 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015620 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.015673 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015636 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.015833 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015816 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.015893 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.015833 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.016445 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016421 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.016528 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016449 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.016691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016673 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.016729 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016693 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.016911 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016894 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.017097 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.016912 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.017182 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017163 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.017223 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017183 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.017383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017366 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.017383 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017382 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.017573 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017557 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.017621 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017574 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.017731 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017716 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.017767 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017732 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.017907 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017884 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.017966 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.017908 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.018131 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018117 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.018169 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018132 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.018331 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018315 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.018331 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018330 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.018525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018506 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.018565 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018525 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.018696 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018681 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.018745 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018696 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.018889 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018874 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.018946 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.018889 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.019091 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019076 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.019143 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019091 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.019302 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019284 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.019346 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019302 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.019506 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019490 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.019547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019506 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.019716 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019695 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.019716 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019715 2584 scope.go:117] "RemoveContainer" containerID="4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661" Jul 06 19:35:24.019942 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019914 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661"} err="failed to get container status \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": rpc error: code = NotFound desc = could not find container \"4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661\": container with ID starting with 4737ec1af8f762823514fa954c18a5d904828735dc0419a7a5d8050c12046661 not found: ID does not exist" Jul 06 19:35:24.019995 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.019945 2584 scope.go:117] "RemoveContainer" containerID="6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243" Jul 06 19:35:24.020150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020134 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243"} err="failed to get container status \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": rpc error: code = NotFound desc = could not find container \"6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243\": container with ID starting with 6eae3087fc7d708531d5a37bf45b8d4238d6b083cc37d96cdafe781f53d68243 not found: ID does not exist" Jul 06 19:35:24.020187 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020150 2584 scope.go:117] "RemoveContainer" containerID="4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe" Jul 06 19:35:24.020379 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020357 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe"} err="failed to get container status \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": rpc error: code = NotFound desc = could not find container \"4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe\": container with ID starting with 4ae47bc7f4ae78f52d1e0dbb9d05403371312ac2619697465dc27eb817d3b9fe not found: ID does not exist" Jul 06 19:35:24.020379 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020377 2584 scope.go:117] "RemoveContainer" containerID="001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b" Jul 06 19:35:24.020571 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020556 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b"} err="failed to get container status \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": rpc error: code = NotFound desc = could not find container \"001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b\": container with ID starting with 001a9930dacd0ae5025731132d4e777e69b468e1685f786ad027ebbf7449924b not found: ID does not exist" Jul 06 19:35:24.020620 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020571 2584 scope.go:117] "RemoveContainer" containerID="2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e" Jul 06 19:35:24.020773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020757 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e"} err="failed to get container status \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": rpc error: code = NotFound desc = could not find container \"2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e\": container with ID starting with 2d00792c032676b613005e723e35eee92d6912e0f158e22a1c918dfd44c42a2e not found: ID does not exist" Jul 06 19:35:24.020773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.020772 2584 scope.go:117] "RemoveContainer" containerID="8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060" Jul 06 19:35:24.021109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.021090 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060"} err="failed to get container status \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": rpc error: code = NotFound desc = could not find container \"8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060\": container with ID starting with 8cf2fa74b36192e171a06123a5d807df69f857d0de6f4c393c744f97fec30060 not found: ID does not exist" Jul 06 19:35:24.021109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.021109 2584 scope.go:117] "RemoveContainer" containerID="3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56" Jul 06 19:35:24.021326 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.021299 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56"} err="failed to get container status \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": rpc error: code = NotFound desc = could not find container \"3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56\": container with ID starting with 3da9918d450ee127e7f352e9d5dc2e9963e427e88ac745a2642a6733a7ddfe56 not found: ID does not exist" Jul 06 19:35:24.036243 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036213 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:24.036469 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036458 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-thanos" Jul 06 19:35:24.036507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036471 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-thanos" Jul 06 19:35:24.036507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036479 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="thanos-sidecar" Jul 06 19:35:24.036507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036485 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="thanos-sidecar" Jul 06 19:35:24.036507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036492 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="init-config-reloader" Jul 06 19:35:24.036507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036498 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="init-config-reloader" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036509 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="prometheus" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036515 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="prometheus" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036522 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="config-reloader" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036527 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="config-reloader" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036532 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-web" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036537 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-web" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036546 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036552 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036596 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-web" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036603 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="config-reloader" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036608 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036613 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="thanos-sidecar" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036618 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="prometheus" Jul 06 19:35:24.036663 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.036622 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" containerName="kube-rbac-proxy-thanos" Jul 06 19:35:24.040156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.040142 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.043483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.043464 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:35:24.043483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.043464 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:35:24.043619 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.043464 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ba1todjcu3ft\"" Jul 06 19:35:24.043619 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.043469 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:35:24.044347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.044334 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:35:24.044660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.044644 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:35:24.044877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.044863 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:35:24.045172 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045156 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:35:24.045675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045551 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:35:24.045675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045611 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:35:24.045675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045655 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:35:24.045879 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045608 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:35:24.045879 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.045658 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nnq86\"" Jul 06 19:35:24.046102 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.046087 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:35:24.049608 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.049592 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:35:24.051409 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.051394 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:35:24.063226 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.063203 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:24.116541 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116501 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116541 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116538 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116558 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116579 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116643 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116665 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116682 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116725 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116767 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kld\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-kube-api-access-54kld\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116787 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116803 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116824 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.116890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116878 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116901 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116960 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.116995 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.117011 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.117041 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.117121 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.117077 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.217823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217731 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.217823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217768 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.217823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217789 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.217823 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217807 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217829 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217851 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217878 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217904 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.217999 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218021 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218045 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218072 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kld\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-kube-api-access-54kld\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218124 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218202 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218230 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218296 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218384 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.218614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.218543 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.219114 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.219084 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.219295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.219276 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.219744 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.219717 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.220390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.220367 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.220722 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.220699 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.220891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.220858 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.221225 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.221201 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.221311 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.221289 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.221369 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.221310 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.221906 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.221827 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.222453 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.222431 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.222755 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.222727 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.222988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.222965 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-config\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.223509 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.223487 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.223658 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.223640 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.223726 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.223684 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.224096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.224080 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.227906 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.227890 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kld\" (UniqueName: \"kubernetes.io/projected/9fdeb009-3b9b-490f-b0a3-146aca6f31cc-kube-api-access-54kld\") pod \"prometheus-k8s-0\" (UID: \"9fdeb009-3b9b-490f-b0a3-146aca6f31cc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.349174 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.349136 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:24.485239 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.485163 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:35:24.488254 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:35:24.488230 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdeb009_3b9b_490f_b0a3_146aca6f31cc.slice/crio-b8d38120fcc33e6a30f6b714a5f67aa620cea7d31cca903bb8d730071ed87dbd WatchSource:0}: Error finding container b8d38120fcc33e6a30f6b714a5f67aa620cea7d31cca903bb8d730071ed87dbd: Status 404 returned error can't find the container with id b8d38120fcc33e6a30f6b714a5f67aa620cea7d31cca903bb8d730071ed87dbd Jul 06 19:35:24.965099 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.965066 2584 generic.go:356] "Generic (PLEG): container finished" podID="9fdeb009-3b9b-490f-b0a3-146aca6f31cc" containerID="7a6979ee02aaf23fe3453abed47feb106aaaf70c12e679099b68d5edc649cde9" exitCode=0 Jul 06 19:35:24.965503 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.965105 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerDied","Data":"7a6979ee02aaf23fe3453abed47feb106aaaf70c12e679099b68d5edc649cde9"} Jul 06 19:35:24.965503 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:24.965153 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"b8d38120fcc33e6a30f6b714a5f67aa620cea7d31cca903bb8d730071ed87dbd"} Jul 06 19:35:25.224613 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.224579 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b98b92-6b84-4912-a3dd-1fff57bdb0d3" path="/var/lib/kubelet/pods/12b98b92-6b84-4912-a3dd-1fff57bdb0d3/volumes" Jul 06 19:35:25.973378 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"3ba5bd4e3b1046c5bac2fbf038f6fc6a24f655e8236ac7ab881452471aa1da78"} Jul 06 19:35:25.973839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973483 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"149efb6091e243597fc3fdded56fe5488c5fda2520047e00035f20c0c46d4c8a"} Jul 06 19:35:25.973839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973545 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"842df92805ac99c8ce9fce6aa9599a40ff537b65345d5b4d546996f593e65116"} Jul 06 19:35:25.973839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973598 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"461fa02c06993f50763c52513e13fe67ddfc4190ec25302ad351d758b0bc2479"} Jul 06 19:35:25.973839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973654 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"42eafa31906450c9c5f73293e880ef1c0bd6a453bd6f9fc522740e2b7486fc1d"} Jul 06 19:35:25.973839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:25.973704 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9fdeb009-3b9b-490f-b0a3-146aca6f31cc","Type":"ContainerStarted","Data":"ee7bda84b9e23f73576dd832fcf0ac05c6fc2437dc00b68ffab7c3db3ad4a312"} Jul 06 19:35:26.002497 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:26.002421 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.983914473 podStartE2EDuration="2.002405869s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.491396ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:26.001647239 +0000 UTC m=+161.349290413" watchObservedRunningTime="2026-07-06 19:35:26.002405869 +0000 UTC m=+161.350049045" Jul 06 19:35:28.050884 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:28.050852 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:28.115518 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:28.115478 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:29.349776 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:29.349737 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:35:38.524209 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:38.524135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:38.624078 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:38.624045 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:38.689819 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:38.689789 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:47.519166 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:47.519116 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:57.137258 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:35:57.137221 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:10.698590 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:10.698556 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.871893 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:22.871854 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:23.746131 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:23.746093 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.349291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:24.349259 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:36:24.364251 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:24.364228 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:36:25.202360 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:25.202335 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:36:33.290087 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:33.290050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:33.349839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:33.349808 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:45.308221 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:45.308187 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:45.373627 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:45.373589 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:48.658298 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:48.658262 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:48.758867 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:48.758838 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:49.466785 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:49.466753 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:49.524135 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:36:49.524101 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:00.791021 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:00.790981 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:00.895270 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:00.895234 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:00.993010 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:00.992960 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:11.697971 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:11.697937 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:11.800568 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:11.800534 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:43.638712 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:43.638680 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:45.042791 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:45.042756 2584 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:37:45.058759 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:45.058728 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:47.291215 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:47.291178 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:47.391737 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:47.391677 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:52.050409 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:52.050368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:52.113842 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:37:52.113809 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:01.021575 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:01.021541 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:01.133144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:01.133110 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:01.312636 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:01.312543 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:01.455610 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:01.455576 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:03.260885 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:03.260842 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:03.445042 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:03.444992 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:11.261085 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:11.261048 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:13.716168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:13.716133 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:13.780160 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:13.780120 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:15.052337 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:15.052300 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:15.153083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:15.153050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:19.704778 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:19.704742 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:19.804891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:19.804860 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:21.702302 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:21.702267 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.472033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:37.471945 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.573529 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:37.573481 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.080024 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.079988 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.406472 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.406391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.517330 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.517299 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.588775 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.588742 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5"] Jul 06 19:38:38.592215 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.592199 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.596373 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.596342 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:38:38.596373 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.596360 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:38:38.596548 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.596381 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:38:38.596966 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.596921 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jt7ld\"" Jul 06 19:38:38.598213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.598195 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:38:38.628589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.628563 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5"] Jul 06 19:38:38.692906 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.692869 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crlr\" (UniqueName: \"kubernetes.io/projected/62d30bf8-8393-4f51-a745-66b1b35078bc-kube-api-access-9crlr\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.693111 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.692912 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.693111 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.692977 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-manifests\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.693111 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.693007 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-tmp\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.693227 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.693107 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.739522 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.739493 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.793690 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.793658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.793863 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.793698 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crlr\" (UniqueName: \"kubernetes.io/projected/62d30bf8-8393-4f51-a745-66b1b35078bc-kube-api-access-9crlr\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.793863 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.793732 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.793863 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.793761 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-manifests\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.793863 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.793793 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-tmp\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.794266 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.794246 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-manifests\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.796158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.796134 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62d30bf8-8393-4f51-a745-66b1b35078bc-tmp\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.796318 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.796297 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.796439 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.796419 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62d30bf8-8393-4f51-a745-66b1b35078bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.809756 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.809725 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crlr\" (UniqueName: \"kubernetes.io/projected/62d30bf8-8393-4f51-a745-66b1b35078bc-kube-api-access-9crlr\") pod \"opendatahub-operator-controller-manager-5cbdc6848b-tb7x5\" (UID: \"62d30bf8-8393-4f51-a745-66b1b35078bc\") " pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.828691 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.828657 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:38.902675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.902648 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:38.930784 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:38.930754 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:39.023964 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:39.023920 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:39.035825 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:39.035796 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5"] Jul 06 19:38:39.038833 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:38:39.038808 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d30bf8_8393_4f51_a745_66b1b35078bc.slice/crio-80dba4ce8da194b048cff4864c328d810a7a04b71ccd27adce5551a7697e2789 WatchSource:0}: Error finding container 80dba4ce8da194b048cff4864c328d810a7a04b71ccd27adce5551a7697e2789: Status 404 returned error can't find the container with id 80dba4ce8da194b048cff4864c328d810a7a04b71ccd27adce5551a7697e2789 Jul 06 19:38:39.040530 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:39.040513 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:38:39.126543 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:39.126511 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:39.657813 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:39.657777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerStarted","Data":"80dba4ce8da194b048cff4864c328d810a7a04b71ccd27adce5551a7697e2789"} Jul 06 19:38:42.676252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:42.676212 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerStarted","Data":"64c64fbbaac5fd17d99d2154c571b76119b7b98b6ba202b2e7c6ae95e87b7c6a"} Jul 06 19:38:43.680779 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:43.680744 2584 generic.go:356] "Generic (PLEG): container finished" podID="62d30bf8-8393-4f51-a745-66b1b35078bc" containerID="64c64fbbaac5fd17d99d2154c571b76119b7b98b6ba202b2e7c6ae95e87b7c6a" exitCode=0 Jul 06 19:38:43.681276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:43.680802 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerDied","Data":"64c64fbbaac5fd17d99d2154c571b76119b7b98b6ba202b2e7c6ae95e87b7c6a"} Jul 06 19:38:44.687817 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:44.687780 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerStarted","Data":"1f2744dfc36d53b724b6758aaf893c0d3f1f6a8b1d37e3bc8d150b261cb2b866"} Jul 06 19:38:44.688241 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:44.688018 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:44.713003 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:44.712947 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" podStartSLOduration=2.459639957 podStartE2EDuration="6.712905715s" totalImagesPullingTime="3.243243322s" totalInitContainerRuntime="1.010022436s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:44.710995565 +0000 UTC m=+360.058638740" watchObservedRunningTime="2026-07-06 19:38:44.712905715 +0000 UTC m=+360.060548889" Jul 06 19:38:47.995365 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:47.995319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:48.059945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:48.059888 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:55.695962 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:55.695921 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:38:58.119532 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:58.119501 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:58.229346 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:58.229309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:58.714441 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:58.714391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:58.954123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:58.954088 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.120427 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:59.120339 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.226954 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:59.226902 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.313905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:59.313865 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.542323 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:38:59.542293 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.175998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:00.175969 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.311208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:00.311158 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.418019 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:00.417983 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:01.654845 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:01.654813 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:05.095418 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.095385 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:05.188965 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.188909 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:05.874909 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.874873 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp"] Jul 06 19:39:05.886198 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.886171 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:05.888814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.888551 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:39:05.888814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.888586 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:39:05.888814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.888630 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:39:05.888814 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.888719 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jjpdq\"" Jul 06 19:39:05.895124 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.895102 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp"] Jul 06 19:39:05.926115 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:05.926089 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:06.014748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.014704 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ftv\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-kube-api-access-66ftv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.014913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.014756 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.014913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.014783 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.014913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.014843 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.014913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.014904 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.015159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.015048 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.015159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.015098 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.015159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.015141 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.015274 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.015182 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.093388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.093316 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:06.116284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116249 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116288 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116312 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ftv\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-kube-api-access-66ftv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116458 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116527 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.116787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116618 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.117120 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116905 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.117120 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116964 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.117120 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.116996 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.117120 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.117096 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.117270 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.117096 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.119195 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.119171 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.119356 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.119338 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.127334 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.127279 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.129780 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.129759 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ftv\" (UniqueName: \"kubernetes.io/projected/2780cfdc-2967-4e12-b65f-494ff2dc99d8-kube-api-access-66ftv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bzclp\" (UID: \"2780cfdc-2967-4e12-b65f-494ff2dc99d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.197748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.197718 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:06.216776 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.216741 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:06.353631 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.353599 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp"] Jul 06 19:39:06.356674 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:39:06.356651 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2780cfdc_2967_4e12_b65f_494ff2dc99d8.slice/crio-2131447388c39e49474f1fdf69ebfe85444d716d0b3b3a7848d6e599418da917 WatchSource:0}: Error finding container 2131447388c39e49474f1fdf69ebfe85444d716d0b3b3a7848d6e599418da917: Status 404 returned error can't find the container with id 2131447388c39e49474f1fdf69ebfe85444d716d0b3b3a7848d6e599418da917 Jul 06 19:39:06.772614 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:06.772579 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" event={"ID":"2780cfdc-2967-4e12-b65f-494ff2dc99d8","Type":"ContainerStarted","Data":"2131447388c39e49474f1fdf69ebfe85444d716d0b3b3a7848d6e599418da917"} Jul 06 19:39:07.636390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.636356 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b"] Jul 06 19:39:07.639028 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.639012 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.641292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.641272 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:39:07.641862 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.641840 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:39:07.642006 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.641842 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:39:07.642006 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.641871 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:39:07.642228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.642211 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mxf8b\"" Jul 06 19:39:07.642329 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.642311 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:39:07.650974 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.650863 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b"] Jul 06 19:39:07.729446 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.729414 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc40740a-3c88-4259-94f3-4bad03ac345d-manager-config\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.729647 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.729511 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmwl\" (UniqueName: \"kubernetes.io/projected/cc40740a-3c88-4259-94f3-4bad03ac345d-kube-api-access-vzmwl\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.729647 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.729561 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-metrics-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.729647 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.729632 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.830273 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.830243 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc40740a-3c88-4259-94f3-4bad03ac345d-manager-config\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.830433 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.830291 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmwl\" (UniqueName: \"kubernetes.io/projected/cc40740a-3c88-4259-94f3-4bad03ac345d-kube-api-access-vzmwl\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.830502 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.830477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-metrics-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.830600 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.830584 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.831045 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.831027 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc40740a-3c88-4259-94f3-4bad03ac345d-manager-config\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.832816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.832783 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-metrics-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.832950 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.832916 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc40740a-3c88-4259-94f3-4bad03ac345d-cert\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.840322 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.840294 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmwl\" (UniqueName: \"kubernetes.io/projected/cc40740a-3c88-4259-94f3-4bad03ac345d-kube-api-access-vzmwl\") pod \"lws-controller-manager-59bc67679f-cw55b\" (UID: \"cc40740a-3c88-4259-94f3-4bad03ac345d\") " pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:07.954042 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:07.954005 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:08.097759 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.097610 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b"] Jul 06 19:39:08.101050 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:39:08.101009 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc40740a_3c88_4259_94f3_4bad03ac345d.slice/crio-7cc14544474014e39f79a99d8bd3abed20b1bb4948b9d30031014ee6abf86d69 WatchSource:0}: Error finding container 7cc14544474014e39f79a99d8bd3abed20b1bb4948b9d30031014ee6abf86d69: Status 404 returned error can't find the container with id 7cc14544474014e39f79a99d8bd3abed20b1bb4948b9d30031014ee6abf86d69 Jul 06 19:39:08.743373 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.743324 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:08.785989 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.785956 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" event={"ID":"cc40740a-3c88-4259-94f3-4bad03ac345d","Type":"ContainerStarted","Data":"7cc14544474014e39f79a99d8bd3abed20b1bb4948b9d30031014ee6abf86d69"} Jul 06 19:39:08.834198 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.834156 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9"] Jul 06 19:39:08.838033 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.838008 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:08.841442 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.841410 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:39:08.841694 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.841676 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fdpl9\"" Jul 06 19:39:08.857680 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.857650 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9"] Jul 06 19:39:08.907165 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.907132 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:08.939674 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.939634 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsjb\" (UniqueName: \"kubernetes.io/projected/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-kube-api-access-grsjb\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:08.939674 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:08.939688 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.041094 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.040996 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsjb\" (UniqueName: \"kubernetes.io/projected/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-kube-api-access-grsjb\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.041094 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.041066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.041310 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:39:09.041213 2584 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 19:39:09.041310 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:39:09.041291 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert podName:2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3 nodeName:}" failed. No retries permitted until 2026-07-06 19:39:09.541271063 +0000 UTC m=+384.888914227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert") pod "kserve-controller-manager-d6f4b65b8-zpjb9" (UID: "2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3") : secret "kserve-webhook-server-cert" not found Jul 06 19:39:09.049466 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.049431 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:09.054755 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.054731 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsjb\" (UniqueName: \"kubernetes.io/projected/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-kube-api-access-grsjb\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.251248 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.251217 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:09.545861 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.545825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.548489 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.548456 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpjb9\" (UID: \"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:09.752274 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:09.752241 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:10.152581 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:10.152531 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:10.256970 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:10.256917 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:11.623194 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.623108 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:39:11.623653 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.623209 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:39:11.623653 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.623260 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:39:11.748392 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.748363 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9"] Jul 06 19:39:11.750726 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:39:11.750698 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2086bdc0_0aa4_4b9f_a1c9_a86274d6b4a3.slice/crio-86526f263e8d3ed8b0386242549871d12e55b95d24250c0f8f0821fea82946df WatchSource:0}: Error finding container 86526f263e8d3ed8b0386242549871d12e55b95d24250c0f8f0821fea82946df: Status 404 returned error can't find the container with id 86526f263e8d3ed8b0386242549871d12e55b95d24250c0f8f0821fea82946df Jul 06 19:39:11.802806 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.802235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" event={"ID":"2780cfdc-2967-4e12-b65f-494ff2dc99d8","Type":"ContainerStarted","Data":"41b08a95ba50ef96c7a268e9b0e9abc273ba6a7c850c282bcc8357af60fd42eb"} Jul 06 19:39:11.804436 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.804405 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" event={"ID":"cc40740a-3c88-4259-94f3-4bad03ac345d","Type":"ContainerStarted","Data":"72a9667c5ae53d1d0443f38d96aa68a678b0edf2b17353ceffae7ea498824b34"} Jul 06 19:39:11.804660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.804507 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:11.805647 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.805626 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" event={"ID":"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3","Type":"ContainerStarted","Data":"86526f263e8d3ed8b0386242549871d12e55b95d24250c0f8f0821fea82946df"} Jul 06 19:39:11.833285 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.833222 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" podStartSLOduration=1.568958394 podStartE2EDuration="6.833198711s" totalImagesPullingTime="5.264240317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:11.831662081 +0000 UTC m=+387.179305257" watchObservedRunningTime="2026-07-06 19:39:11.833198711 +0000 UTC m=+387.180841887" Jul 06 19:39:11.858359 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:11.858311 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" podStartSLOduration=1.278483649 podStartE2EDuration="4.85829219s" totalImagesPullingTime="3.579808541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:11.856214642 +0000 UTC m=+387.203857834" watchObservedRunningTime="2026-07-06 19:39:11.85829219 +0000 UTC m=+387.205935364" Jul 06 19:39:12.198867 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:12.198837 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:12.203214 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:12.203189 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:12.810148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:12.810113 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:12.811100 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:12.811079 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bzclp" Jul 06 19:39:14.821213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:14.821177 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" event={"ID":"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3","Type":"ContainerStarted","Data":"7bc9a61ad90dee43e438112d14d1d414e7cb278607d979568254e5d9ee3d78ee"} Jul 06 19:39:14.821660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:14.821258 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:14.849457 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:14.849402 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" podStartSLOduration=4.256041923 podStartE2EDuration="6.849385131s" totalImagesPullingTime="2.593343208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:14.847518105 +0000 UTC m=+390.195161282" watchObservedRunningTime="2026-07-06 19:39:14.849385131 +0000 UTC m=+390.197028305" Jul 06 19:39:20.209850 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:20.209811 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:20.279379 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:20.279343 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:22.812064 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:22.812038 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59bc67679f-cw55b" Jul 06 19:39:33.971123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:33.971081 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:36.326275 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:36.326244 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:36.386214 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:36.386175 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:45.831833 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:45.831801 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:39:59.714354 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.714315 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:59.816946 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.816899 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:59.861200 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.861159 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:39:59.865462 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.865440 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:39:59.869658 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.869637 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:39:59.869984 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.869720 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sbttr\"" Jul 06 19:39:59.869984 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.869775 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:39:59.876559 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.876527 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:39:59.952865 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.952830 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:59.976101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:39:59.976022 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfph\" (UniqueName: \"kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph\") pod \"kuadrant-operator-catalog-nn9mx\" (UID: \"5372aefd-c529-4a33-b9d1-5ccca568a5f9\") " pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:00.077158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.077122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfph\" (UniqueName: \"kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph\") pod \"kuadrant-operator-catalog-nn9mx\" (UID: \"5372aefd-c529-4a33-b9d1-5ccca568a5f9\") " pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:00.085116 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.085091 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfph\" (UniqueName: \"kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph\") pod \"kuadrant-operator-catalog-nn9mx\" (UID: \"5372aefd-c529-4a33-b9d1-5ccca568a5f9\") " pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:00.176326 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.176279 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:00.218015 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.217987 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:40:00.305349 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.305317 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:40:00.308874 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:40:00.308848 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5372aefd_c529_4a33_b9d1_5ccca568a5f9.slice/crio-f81387d43d3906f0992e5009c688cfc5f713f85fce2af1ad0c3ee327bcef4ebd WatchSource:0}: Error finding container f81387d43d3906f0992e5009c688cfc5f713f85fce2af1ad0c3ee327bcef4ebd: Status 404 returned error can't find the container with id f81387d43d3906f0992e5009c688cfc5f713f85fce2af1ad0c3ee327bcef4ebd Jul 06 19:40:00.426864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.426829 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z8zcq"] Jul 06 19:40:00.429392 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.429373 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:00.438030 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.438005 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z8zcq"] Jul 06 19:40:00.480522 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.480495 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpcp\" (UniqueName: \"kubernetes.io/projected/9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a-kube-api-access-4bpcp\") pod \"kuadrant-operator-catalog-z8zcq\" (UID: \"9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a\") " pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:00.581646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.581574 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpcp\" (UniqueName: \"kubernetes.io/projected/9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a-kube-api-access-4bpcp\") pod \"kuadrant-operator-catalog-z8zcq\" (UID: \"9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a\") " pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:00.592408 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.592380 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpcp\" (UniqueName: \"kubernetes.io/projected/9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a-kube-api-access-4bpcp\") pod \"kuadrant-operator-catalog-z8zcq\" (UID: \"9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a\") " pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:00.739429 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.739386 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:00.861994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:00.861744 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z8zcq"] Jul 06 19:40:00.864765 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:40:00.864740 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e37dc3a_11e0_4261_a6a3_6cbce0bc2c4a.slice/crio-f2a8214235eca27a712d7198c5404be5a65408ac5e5a460da1ff970829a8e968 WatchSource:0}: Error finding container f2a8214235eca27a712d7198c5404be5a65408ac5e5a460da1ff970829a8e968: Status 404 returned error can't find the container with id f2a8214235eca27a712d7198c5404be5a65408ac5e5a460da1ff970829a8e968 Jul 06 19:40:01.013809 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:01.013773 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" event={"ID":"9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a","Type":"ContainerStarted","Data":"f2a8214235eca27a712d7198c5404be5a65408ac5e5a460da1ff970829a8e968"} Jul 06 19:40:01.014895 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:01.014870 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" event={"ID":"5372aefd-c529-4a33-b9d1-5ccca568a5f9","Type":"ContainerStarted","Data":"f81387d43d3906f0992e5009c688cfc5f713f85fce2af1ad0c3ee327bcef4ebd"} Jul 06 19:40:03.026762 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.026662 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" event={"ID":"5372aefd-c529-4a33-b9d1-5ccca568a5f9","Type":"ContainerStarted","Data":"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334"} Jul 06 19:40:03.027226 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.026729 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" podUID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" containerName="registry-server" containerID="cri-o://f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334" gracePeriod=2 Jul 06 19:40:03.028167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.028142 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" event={"ID":"9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a","Type":"ContainerStarted","Data":"9a789ff7ab22b2a18e6674f668a776f7432ea679c36bd491f4fd73b834b6db14"} Jul 06 19:40:03.042908 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.042853 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" podStartSLOduration=1.577554647 podStartE2EDuration="4.042833674s" totalImagesPullingTime="2.465279027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:03.042246239 +0000 UTC m=+438.389889414" watchObservedRunningTime="2026-07-06 19:40:03.042833674 +0000 UTC m=+438.390476849" Jul 06 19:40:03.061564 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.061511 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" podStartSLOduration=1.133300726 podStartE2EDuration="3.061495917s" totalImagesPullingTime="1.928195191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:03.060388867 +0000 UTC m=+438.408032044" watchObservedRunningTime="2026-07-06 19:40:03.061495917 +0000 UTC m=+438.409139091" Jul 06 19:40:03.270349 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.270325 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:03.409440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.409352 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmfph\" (UniqueName: \"kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph\") pod \"5372aefd-c529-4a33-b9d1-5ccca568a5f9\" (UID: \"5372aefd-c529-4a33-b9d1-5ccca568a5f9\") " Jul 06 19:40:03.411597 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.411566 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph" (OuterVolumeSpecName: "kube-api-access-zmfph") pod "5372aefd-c529-4a33-b9d1-5ccca568a5f9" (UID: "5372aefd-c529-4a33-b9d1-5ccca568a5f9"). InnerVolumeSpecName "kube-api-access-zmfph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:40:03.510150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:03.510107 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfph\" (UniqueName: \"kubernetes.io/projected/5372aefd-c529-4a33-b9d1-5ccca568a5f9-kube-api-access-zmfph\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:40:04.034267 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.034227 2584 generic.go:356] "Generic (PLEG): container finished" podID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" containerID="f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334" exitCode=0 Jul 06 19:40:04.034267 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.034269 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" Jul 06 19:40:04.034771 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.034319 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" event={"ID":"5372aefd-c529-4a33-b9d1-5ccca568a5f9","Type":"ContainerDied","Data":"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334"} Jul 06 19:40:04.034771 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.034374 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nn9mx" event={"ID":"5372aefd-c529-4a33-b9d1-5ccca568a5f9","Type":"ContainerDied","Data":"f81387d43d3906f0992e5009c688cfc5f713f85fce2af1ad0c3ee327bcef4ebd"} Jul 06 19:40:04.034771 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.034407 2584 scope.go:117] "RemoveContainer" containerID="f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334" Jul 06 19:40:04.048808 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.048765 2584 scope.go:117] "RemoveContainer" containerID="f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334" Jul 06 19:40:04.049336 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:40:04.049314 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334\": container with ID starting with f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334 not found: ID does not exist" containerID="f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334" Jul 06 19:40:04.049428 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.049342 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334"} err="failed to get container status \"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334\": rpc error: code = NotFound desc = could not find container \"f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334\": container with ID starting with f98f54e91fa06ba11cd619d68f97d52ecd518f97b42bbdb6fa736413f1793334 not found: ID does not exist" Jul 06 19:40:04.059834 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.059765 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:40:04.063588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:04.063563 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nn9mx"] Jul 06 19:40:05.223352 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:05.223321 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" path="/var/lib/kubelet/pods/5372aefd-c529-4a33-b9d1-5ccca568a5f9/volumes" Jul 06 19:40:10.740335 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:10.740299 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:10.740745 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:10.740358 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:10.762369 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:10.762344 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:11.086812 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:11.086743 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-z8zcq" Jul 06 19:40:14.394530 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:14.394500 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:14.456242 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:14.456209 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:23.115832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:23.115792 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:24.919669 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:24.919634 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:27.108071 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.108030 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8"] Jul 06 19:40:27.108444 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.108400 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" containerName="registry-server" Jul 06 19:40:27.108444 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.108413 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" containerName="registry-server" Jul 06 19:40:27.108522 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.108474 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5372aefd-c529-4a33-b9d1-5ccca568a5f9" containerName="registry-server" Jul 06 19:40:27.111559 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.111541 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:27.114321 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.114301 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dqjpq\"" Jul 06 19:40:27.129733 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.129700 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8"] Jul 06 19:40:27.211493 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.211454 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmkq\" (UniqueName: \"kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq\") pod \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" (UID: \"80968e72-bc12-4a1e-b083-0f7b106a8016\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:27.312191 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.312157 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmkq\" (UniqueName: \"kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq\") pod \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" (UID: \"80968e72-bc12-4a1e-b083-0f7b106a8016\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:27.323338 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.323304 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmkq\" (UniqueName: \"kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq\") pod \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" (UID: \"80968e72-bc12-4a1e-b083-0f7b106a8016\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:27.421936 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.421901 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:27.555270 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.555243 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8"] Jul 06 19:40:27.557634 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:40:27.557609 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80968e72_bc12_4a1e_b083_0f7b106a8016.slice/crio-257ff29347dedd7f7fe9d5645ba6ee41f249c5698fa3f41900322c94f39a82c2 WatchSource:0}: Error finding container 257ff29347dedd7f7fe9d5645ba6ee41f249c5698fa3f41900322c94f39a82c2: Status 404 returned error can't find the container with id 257ff29347dedd7f7fe9d5645ba6ee41f249c5698fa3f41900322c94f39a82c2 Jul 06 19:40:27.717052 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:27.716972 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:28.137335 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:28.137253 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" event={"ID":"80968e72-bc12-4a1e-b083-0f7b106a8016","Type":"ContainerStarted","Data":"257ff29347dedd7f7fe9d5645ba6ee41f249c5698fa3f41900322c94f39a82c2"} Jul 06 19:40:28.568870 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:28.568813 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:30.146432 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:30.146398 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" event={"ID":"80968e72-bc12-4a1e-b083-0f7b106a8016","Type":"ContainerStarted","Data":"bae4561f89848458588b995fe079360fc9d651e16fc56b7fe5d0874272b3eb15"} Jul 06 19:40:30.146871 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:30.146514 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:30.167508 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:30.167454 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" podStartSLOduration=2.591304798 podStartE2EDuration="4.167437033s" totalImagesPullingTime="1.576132235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:30.166256393 +0000 UTC m=+465.513899569" watchObservedRunningTime="2026-07-06 19:40:30.167437033 +0000 UTC m=+465.515080208" Jul 06 19:40:32.385474 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:32.385443 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:32.573922 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:32.573884 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:34.379059 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.379028 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:34.709956 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.709913 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8"] Jul 06 19:40:34.714284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.714268 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:34.717434 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.717413 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2gj4c\"" Jul 06 19:40:34.717919 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.717900 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:40:34.732423 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.732404 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8"] Jul 06 19:40:34.779076 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.779050 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmhl\" (UniqueName: \"kubernetes.io/projected/b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e-kube-api-access-bsmhl\") pod \"dns-operator-controller-manager-864dd78c4c-llhq8\" (UID: \"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:34.880351 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.880318 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmhl\" (UniqueName: \"kubernetes.io/projected/b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e-kube-api-access-bsmhl\") pod \"dns-operator-controller-manager-864dd78c4c-llhq8\" (UID: \"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:34.892526 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:34.892494 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmhl\" (UniqueName: \"kubernetes.io/projected/b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e-kube-api-access-bsmhl\") pod \"dns-operator-controller-manager-864dd78c4c-llhq8\" (UID: \"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:35.024148 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:35.024067 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:35.168432 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:35.168411 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8"] Jul 06 19:40:35.170793 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:40:35.170761 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f944e6_bd46_4b41_9e7c_9d7ba0dae93e.slice/crio-e0d00644217d07e1d1b958e92f5b78fa42782254d04c7d481fa9a26ff15907cc WatchSource:0}: Error finding container e0d00644217d07e1d1b958e92f5b78fa42782254d04c7d481fa9a26ff15907cc: Status 404 returned error can't find the container with id e0d00644217d07e1d1b958e92f5b78fa42782254d04c7d481fa9a26ff15907cc Jul 06 19:40:35.333799 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:35.333698 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:35.754822 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:35.754779 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:36.174767 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:36.174737 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" event={"ID":"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e","Type":"ContainerStarted","Data":"e0d00644217d07e1d1b958e92f5b78fa42782254d04c7d481fa9a26ff15907cc"} Jul 06 19:40:38.183660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:38.183629 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" event={"ID":"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e","Type":"ContainerStarted","Data":"854df3de3d5f3cc2d0cfb1271e9b4788ac411099df36a2b964098342a88a64bc"} Jul 06 19:40:38.184123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:38.183726 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:38.203312 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:38.203265 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" podStartSLOduration=2.24525643 podStartE2EDuration="4.203249139s" totalImagesPullingTime="1.957992709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:38.201759655 +0000 UTC m=+473.549402867" watchObservedRunningTime="2026-07-06 19:40:38.203249139 +0000 UTC m=+473.550892342" Jul 06 19:40:41.153166 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:41.153138 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:40:48.769844 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:48.769792 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:49.046547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.046454 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:49.186280 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.186251 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:49.189813 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.189790 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:40:49.568536 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.568499 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm"] Jul 06 19:40:49.573072 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.573052 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.576555 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.576534 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-htwnp\"" Jul 06 19:40:49.591683 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.591657 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm"] Jul 06 19:40:49.718474 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.718430 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.718668 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.718534 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn9h\" (UniqueName: \"kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.819308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.819215 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn9h\" (UniqueName: \"kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.819308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.819296 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.819721 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.819702 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.849547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.849512 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn9h\" (UniqueName: \"kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:49.883080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:49.883031 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:50.029952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:50.029859 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm"] Jul 06 19:40:50.032386 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:40:50.032356 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5e8661_4785_40d1_a79d_a266c9cfaa67.slice/crio-1ba64464b127dd4ed2b4d49fe164f07bc8c57b6e25b71e9f423c5c40e6faa07e WatchSource:0}: Error finding container 1ba64464b127dd4ed2b4d49fe164f07bc8c57b6e25b71e9f423c5c40e6faa07e: Status 404 returned error can't find the container with id 1ba64464b127dd4ed2b4d49fe164f07bc8c57b6e25b71e9f423c5c40e6faa07e Jul 06 19:40:50.236449 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:50.236413 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" event={"ID":"7d5e8661-4785-40d1-a79d-a266c9cfaa67","Type":"ContainerStarted","Data":"1ba64464b127dd4ed2b4d49fe164f07bc8c57b6e25b71e9f423c5c40e6faa07e"} Jul 06 19:40:56.261080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:56.261029 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:56.268436 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:56.268367 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" event={"ID":"7d5e8661-4785-40d1-a79d-a266c9cfaa67","Type":"ContainerStarted","Data":"b210b91aa2a09f535e87ee99d70b5047cba365bb243041335d2ca1060bda971e"} Jul 06 19:40:56.268436 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:56.268440 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:40:56.294715 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:56.294667 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" podStartSLOduration=1.799553596 podStartE2EDuration="7.294651101s" totalImagesPullingTime="5.495097505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:56.293080258 +0000 UTC m=+491.640723434" watchObservedRunningTime="2026-07-06 19:40:56.294651101 +0000 UTC m=+491.642294275" Jul 06 19:40:56.358375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:56.358342 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:58.341493 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:58.341450 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:58.454892 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:58.454855 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:58.888946 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:58.888897 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:58.965772 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:40:58.965731 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:03.225648 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:03.225619 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:03.325430 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:03.325398 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:07.275262 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:07.275235 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:41:08.748426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.748391 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs"] Jul 06 19:41:08.752407 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.752385 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.774123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.774087 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwt24\" (UniqueName: \"kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.774293 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.774189 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.775984 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.775961 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs"] Jul 06 19:41:08.875677 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.875638 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwt24\" (UniqueName: \"kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.875877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.875690 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.876151 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.876131 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:08.884374 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:08.884351 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwt24\" (UniqueName: \"kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24\") pod \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:09.061599 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.061515 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:09.103429 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.103399 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs"] Jul 06 19:41:09.107633 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.107601 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs"] Jul 06 19:41:09.149386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.149358 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:09.153878 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.153861 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.164268 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.164244 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm"] Jul 06 19:41:09.164493 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.164465 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" containerName="manager" containerID="cri-o://b210b91aa2a09f535e87ee99d70b5047cba365bb243041335d2ca1060bda971e" gracePeriod=2 Jul 06 19:41:09.166232 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.166178 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:09.167641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.167618 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm"] Jul 06 19:41:09.178222 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.178187 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ng5\" (UniqueName: \"kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.178348 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:09.178264 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:41:09.178348 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.178285 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.204114 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.204087 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8"] Jul 06 19:41:09.204357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.204311 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" containerName="manager" containerID="cri-o://bae4561f89848458588b995fe079360fc9d651e16fc56b7fe5d0874272b3eb15" gracePeriod=2 Jul 06 19:41:09.206401 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:09.206373 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jdxfm\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:41:09.255430 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:09.255391 2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"80968e72-bc12-4a1e-b083-0f7b106a8016\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-n7cv8\": pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:09.256877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.256851 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8"] Jul 06 19:41:09.279008 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.278978 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ng5\" (UniqueName: \"kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.279172 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.279022 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.279448 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.279414 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.303136 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.303107 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ng5\" (UniqueName: \"kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5\") pod \"kuadrant-operator-controller-manager-55467d9f7-f5pqh\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.327504 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.327474 2584 generic.go:356] "Generic (PLEG): container finished" podID="80968e72-bc12-4a1e-b083-0f7b106a8016" containerID="bae4561f89848458588b995fe079360fc9d651e16fc56b7fe5d0874272b3eb15" exitCode=0 Jul 06 19:41:09.329440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.329414 2584 generic.go:356] "Generic (PLEG): container finished" podID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" containerID="b210b91aa2a09f535e87ee99d70b5047cba365bb243041335d2ca1060bda971e" exitCode=0 Jul 06 19:41:09.425163 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.425139 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:41:09.427085 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:09.427057 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:09.458857 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.458831 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:09.460782 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:09.460758 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:09.467109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.467084 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:09.480902 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.480873 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzmkq\" (UniqueName: \"kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq\") pod \"80968e72-bc12-4a1e-b083-0f7b106a8016\" (UID: \"80968e72-bc12-4a1e-b083-0f7b106a8016\") " Jul 06 19:41:09.481092 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.481064 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume\") pod \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " Jul 06 19:41:09.481145 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.481106 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmn9h\" (UniqueName: \"kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h\") pod \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\" (UID: \"7d5e8661-4785-40d1-a79d-a266c9cfaa67\") " Jul 06 19:41:09.481589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.481561 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7d5e8661-4785-40d1-a79d-a266c9cfaa67" (UID: "7d5e8661-4785-40d1-a79d-a266c9cfaa67"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:09.482821 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.482799 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq" (OuterVolumeSpecName: "kube-api-access-hzmkq") pod "80968e72-bc12-4a1e-b083-0f7b106a8016" (UID: "80968e72-bc12-4a1e-b083-0f7b106a8016"). InnerVolumeSpecName "kube-api-access-hzmkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:09.483123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.483106 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h" (OuterVolumeSpecName: "kube-api-access-pmn9h") pod "7d5e8661-4785-40d1-a79d-a266c9cfaa67" (UID: "7d5e8661-4785-40d1-a79d-a266c9cfaa67"). InnerVolumeSpecName "kube-api-access-pmn9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:09.582074 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.582006 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8661-4785-40d1-a79d-a266c9cfaa67-extensions-socket-volume\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:09.582074 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.582033 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn9h\" (UniqueName: \"kubernetes.io/projected/7d5e8661-4785-40d1-a79d-a266c9cfaa67-kube-api-access-pmn9h\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:09.582074 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.582043 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzmkq\" (UniqueName: \"kubernetes.io/projected/80968e72-bc12-4a1e-b083-0f7b106a8016-kube-api-access-hzmkq\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:09.607897 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.607864 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:09.611188 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:41:09.611162 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977455f8_295c_47eb_9598_28166a297d70.slice/crio-49bf136c42cedb403700baee9a53645172b2359fc83d95f66596c565d6c5e685 WatchSource:0}: Error finding container 49bf136c42cedb403700baee9a53645172b2359fc83d95f66596c565d6c5e685: Status 404 returned error can't find the container with id 49bf136c42cedb403700baee9a53645172b2359fc83d95f66596c565d6c5e685 Jul 06 19:41:09.933160 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:09.933117 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:10.051306 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.051269 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:10.106161 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.106123 2584 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 06 19:41:10.106161 ip-10-0-136-184 kubenswrapper[2584]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-rk5xs_kuadrant-system_e715b6af-2d1d-44e6-9244-fa7ce351b963_0(99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-rk5xs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b" Netns:"/var/run/netns/e10d23bd-49f9-4f9d-9b0d-4519db56ef83" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-rk5xs;K8S_POD_INFRA_CONTAINER_ID=99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b;K8S_POD_UID=e715b6af-2d1d-44e6-9244-fa7ce351b963" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs/e715b6af-2d1d-44e6-9244-fa7ce351b963:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs 99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b network default NAD default NAD key ] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs 99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b network default NAD default NAD key ] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-55467d9f7-rk5xs" not found Jul 06 19:41:10.106161 ip-10-0-136-184 kubenswrapper[2584]: ' Jul 06 19:41:10.106161 ip-10-0-136-184 kubenswrapper[2584]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 19:41:10.106161 ip-10-0-136-184 kubenswrapper[2584]: > Jul 06 19:41:10.106368 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.106194 2584 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 06 19:41:10.106368 ip-10-0-136-184 kubenswrapper[2584]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-rk5xs_kuadrant-system_e715b6af-2d1d-44e6-9244-fa7ce351b963_0(99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-rk5xs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b" Netns:"/var/run/netns/e10d23bd-49f9-4f9d-9b0d-4519db56ef83" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-rk5xs;K8S_POD_INFRA_CONTAINER_ID=99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b;K8S_POD_UID=e715b6af-2d1d-44e6-9244-fa7ce351b963" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs/e715b6af-2d1d-44e6-9244-fa7ce351b963:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs 99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b network default NAD default NAD key ] [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs 99fb9edd610ace4286efbee26db61703da02a04aede0aab27f772a02301c928b network default NAD default NAD key ] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-55467d9f7-rk5xs" not found Jul 06 19:41:10.106368 ip-10-0-136-184 kubenswrapper[2584]: ' Jul 06 19:41:10.106368 ip-10-0-136-184 kubenswrapper[2584]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 06 19:41:10.106368 ip-10-0-136-184 kubenswrapper[2584]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:10.335592 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.335562 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jdxfm" Jul 06 19:41:10.335781 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.335609 2584 scope.go:117] "RemoveContainer" containerID="b210b91aa2a09f535e87ee99d70b5047cba365bb243041335d2ca1060bda971e" Jul 06 19:41:10.336991 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.336967 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:10.337665 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.337631 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:10.338748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.338723 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" event={"ID":"977455f8-295c-47eb-9598-28166a297d70","Type":"ContainerStarted","Data":"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633"} Jul 06 19:41:10.338748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.338740 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:10.338909 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.338786 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" event={"ID":"977455f8-295c-47eb-9598-28166a297d70","Type":"ContainerStarted","Data":"49bf136c42cedb403700baee9a53645172b2359fc83d95f66596c565d6c5e685"} Jul 06 19:41:10.339636 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.339612 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:10.347132 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.347110 2584 scope.go:117] "RemoveContainer" containerID="bae4561f89848458588b995fe079360fc9d651e16fc56b7fe5d0874272b3eb15" Jul 06 19:41:10.347838 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.347823 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:10.382066 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.382018 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" podStartSLOduration=1.382003105 podStartE2EDuration="1.382003105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:10.379451362 +0000 UTC m=+505.727094541" watchObservedRunningTime="2026-07-06 19:41:10.382003105 +0000 UTC m=+505.729646346" Jul 06 19:41:10.382241 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.382077 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:10.386072 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.386048 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="e715b6af-2d1d-44e6-9244-fa7ce351b963" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:10.387896 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:10.387877 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n7cv8\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n7cv8" Jul 06 19:41:10.389169 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.389149 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume\") pod \"e715b6af-2d1d-44e6-9244-fa7ce351b963\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " Jul 06 19:41:10.389258 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.389193 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwt24\" (UniqueName: \"kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24\") pod \"e715b6af-2d1d-44e6-9244-fa7ce351b963\" (UID: \"e715b6af-2d1d-44e6-9244-fa7ce351b963\") " Jul 06 19:41:10.389404 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.389380 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e715b6af-2d1d-44e6-9244-fa7ce351b963" (UID: "e715b6af-2d1d-44e6-9244-fa7ce351b963"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:10.391153 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.391133 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24" (OuterVolumeSpecName: "kube-api-access-gwt24") pod "e715b6af-2d1d-44e6-9244-fa7ce351b963" (UID: "e715b6af-2d1d-44e6-9244-fa7ce351b963"). InnerVolumeSpecName "kube-api-access-gwt24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:10.490547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.490514 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e715b6af-2d1d-44e6-9244-fa7ce351b963-extensions-socket-volume\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:10.490547 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:10.490542 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwt24\" (UniqueName: \"kubernetes.io/projected/e715b6af-2d1d-44e6-9244-fa7ce351b963-kube-api-access-gwt24\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:11.222338 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:11.222305 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" path="/var/lib/kubelet/pods/7d5e8661-4785-40d1-a79d-a266c9cfaa67/volumes" Jul 06 19:41:11.222782 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:11.222602 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" path="/var/lib/kubelet/pods/80968e72-bc12-4a1e-b083-0f7b106a8016/volumes" Jul 06 19:41:11.222940 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:11.222912 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e715b6af-2d1d-44e6-9244-fa7ce351b963" path="/var/lib/kubelet/pods/e715b6af-2d1d-44e6-9244-fa7ce351b963/volumes" Jul 06 19:41:11.344760 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:11.344736 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:11.345034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:11.345017 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:11.348626 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:11.348600 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-rk5xs\" is forbidden: User \"system:node:ip-10-0-136-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-184.ec2.internal' and this object" podUID="e715b6af-2d1d-44e6-9244-fa7ce351b963" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rk5xs" Jul 06 19:41:13.314295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:13.314266 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:13.420516 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:13.420476 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:22.351425 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:22.351395 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:26.178498 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.178454 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:26.334507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.334461 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:26.414685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.414657 2584 generic.go:356] "Generic (PLEG): container finished" podID="2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3" containerID="7bc9a61ad90dee43e438112d14d1d414e7cb278607d979568254e5d9ee3d78ee" exitCode=1 Jul 06 19:41:26.414907 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.414732 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" event={"ID":"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3","Type":"ContainerDied","Data":"7bc9a61ad90dee43e438112d14d1d414e7cb278607d979568254e5d9ee3d78ee"} Jul 06 19:41:26.415164 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.415137 2584 scope.go:117] "RemoveContainer" containerID="7bc9a61ad90dee43e438112d14d1d414e7cb278607d979568254e5d9ee3d78ee" Jul 06 19:41:26.416440 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.416420 2584 generic.go:356] "Generic (PLEG): container finished" podID="b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e" containerID="854df3de3d5f3cc2d0cfb1271e9b4788ac411099df36a2b964098342a88a64bc" exitCode=1 Jul 06 19:41:26.416510 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.416488 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" event={"ID":"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e","Type":"ContainerDied","Data":"854df3de3d5f3cc2d0cfb1271e9b4788ac411099df36a2b964098342a88a64bc"} Jul 06 19:41:26.416839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.416825 2584 scope.go:117] "RemoveContainer" containerID="854df3de3d5f3cc2d0cfb1271e9b4788ac411099df36a2b964098342a88a64bc" Jul 06 19:41:26.418557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.418539 2584 generic.go:356] "Generic (PLEG): container finished" podID="62d30bf8-8393-4f51-a745-66b1b35078bc" containerID="1f2744dfc36d53b724b6758aaf893c0d3f1f6a8b1d37e3bc8d150b261cb2b866" exitCode=1 Jul 06 19:41:26.418661 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.418610 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerDied","Data":"1f2744dfc36d53b724b6758aaf893c0d3f1f6a8b1d37e3bc8d150b261cb2b866"} Jul 06 19:41:26.419057 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.419038 2584 scope.go:117] "RemoveContainer" containerID="1f2744dfc36d53b724b6758aaf893c0d3f1f6a8b1d37e3bc8d150b261cb2b866" Jul 06 19:41:26.473845 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:26.473816 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:27.425134 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.425100 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" event={"ID":"2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3","Type":"ContainerStarted","Data":"badc199989989881304f426067f82f06d85d0bd7152c0cb0c6d61daf8d5f144d"} Jul 06 19:41:27.425601 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.425328 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:41:27.426900 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.426870 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" event={"ID":"b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e","Type":"ContainerStarted","Data":"92484d6b650e60b7a96927945e45a3270b2b0fbf597f27fc7ea0cba58cc3fb82"} Jul 06 19:41:27.427077 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.427061 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:41:27.428836 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.428818 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" event={"ID":"62d30bf8-8393-4f51-a745-66b1b35078bc","Type":"ContainerStarted","Data":"b1f1ec521e8a41c5a3beb5196122a081a5666b71c2b4fd2fb6b42ee5ce08e457"} Jul 06 19:41:27.429069 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:27.429056 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:41:28.274678 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:28.274641 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:29.098552 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098518 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:41:29.098954 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098865 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" containerName="manager" Jul 06 19:41:29.098954 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098876 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" containerName="manager" Jul 06 19:41:29.098954 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098895 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" containerName="manager" Jul 06 19:41:29.098954 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098901 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" containerName="manager" Jul 06 19:41:29.099083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098977 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5e8661-4785-40d1-a79d-a266c9cfaa67" containerName="manager" Jul 06 19:41:29.099083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.098985 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="80968e72-bc12-4a1e-b083-0f7b106a8016" containerName="manager" Jul 06 19:41:29.102458 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.102440 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.123754 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.123711 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:41:29.159023 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.158984 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxcg\" (UniqueName: \"kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.159189 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.159067 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.260324 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.260290 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.260499 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.260357 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxcg\" (UniqueName: \"kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.260664 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.260644 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.270763 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.270730 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxcg\" (UniqueName: \"kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg\") pod \"kuadrant-operator-controller-manager-86878c769f-8f4sh\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.412481 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.412454 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:29.574316 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:29.574275 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:41:29.577689 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:41:29.577660 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756a2264_2fd7_40eb_b097_4fee43934078.slice/crio-97341a35615ad4c0237b4c1e4fcd81b5a800952d57f8d7c9fba2a6108dcfd95c WatchSource:0}: Error finding container 97341a35615ad4c0237b4c1e4fcd81b5a800952d57f8d7c9fba2a6108dcfd95c: Status 404 returned error can't find the container with id 97341a35615ad4c0237b4c1e4fcd81b5a800952d57f8d7c9fba2a6108dcfd95c Jul 06 19:41:30.444857 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.444815 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" event={"ID":"756a2264-2fd7-40eb-b097-4fee43934078","Type":"ContainerStarted","Data":"8c89a3671dca4980ca95d9722201f32e344297fcabd02d87cba9833c4f949ae1"} Jul 06 19:41:30.445317 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.444876 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" event={"ID":"756a2264-2fd7-40eb-b097-4fee43934078","Type":"ContainerStarted","Data":"97341a35615ad4c0237b4c1e4fcd81b5a800952d57f8d7c9fba2a6108dcfd95c"} Jul 06 19:41:30.445317 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.444966 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:30.475248 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.475188 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" podStartSLOduration=1.475165797 podStartE2EDuration="1.475165797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:30.474449405 +0000 UTC m=+525.822092593" watchObservedRunningTime="2026-07-06 19:41:30.475165797 +0000 UTC m=+525.822808973" Jul 06 19:41:30.696573 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.696488 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:30.790588 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:30.790553 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:37.341329 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:37.341249 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:37.454952 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:37.454896 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:38.435896 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:38.435871 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cbdc6848b-tb7x5" Jul 06 19:41:38.436344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:38.435952 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-llhq8" Jul 06 19:41:41.117426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.117391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:41.217997 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.217958 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:41.345889 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.345835 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:41.451344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.451313 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:41:41.452023 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.451996 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:41.534700 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.534666 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:41.534941 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.534903 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" podUID="977455f8-295c-47eb-9598-28166a297d70" containerName="manager" containerID="cri-o://c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633" gracePeriod=10 Jul 06 19:41:41.775461 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.775430 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:41.869687 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.869656 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume\") pod \"977455f8-295c-47eb-9598-28166a297d70\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " Jul 06 19:41:41.869870 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.869716 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22ng5\" (UniqueName: \"kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5\") pod \"977455f8-295c-47eb-9598-28166a297d70\" (UID: \"977455f8-295c-47eb-9598-28166a297d70\") " Jul 06 19:41:41.870080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.870052 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "977455f8-295c-47eb-9598-28166a297d70" (UID: "977455f8-295c-47eb-9598-28166a297d70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:41.871699 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.871674 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5" (OuterVolumeSpecName: "kube-api-access-22ng5") pod "977455f8-295c-47eb-9598-28166a297d70" (UID: "977455f8-295c-47eb-9598-28166a297d70"). InnerVolumeSpecName "kube-api-access-22ng5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:41.970401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.970354 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/977455f8-295c-47eb-9598-28166a297d70-extensions-socket-volume\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:41.970401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:41.970394 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ng5\" (UniqueName: \"kubernetes.io/projected/977455f8-295c-47eb-9598-28166a297d70-kube-api-access-22ng5\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:41:42.498100 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.498067 2584 generic.go:356] "Generic (PLEG): container finished" podID="977455f8-295c-47eb-9598-28166a297d70" containerID="c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633" exitCode=0 Jul 06 19:41:42.498672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.498111 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" Jul 06 19:41:42.498672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.498135 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" event={"ID":"977455f8-295c-47eb-9598-28166a297d70","Type":"ContainerDied","Data":"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633"} Jul 06 19:41:42.498672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.498177 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh" event={"ID":"977455f8-295c-47eb-9598-28166a297d70","Type":"ContainerDied","Data":"49bf136c42cedb403700baee9a53645172b2359fc83d95f66596c565d6c5e685"} Jul 06 19:41:42.498672 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.498199 2584 scope.go:117] "RemoveContainer" containerID="c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633" Jul 06 19:41:42.506466 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.506448 2584 scope.go:117] "RemoveContainer" containerID="c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633" Jul 06 19:41:42.506760 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:41:42.506740 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633\": container with ID starting with c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633 not found: ID does not exist" containerID="c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633" Jul 06 19:41:42.506809 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.506769 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633"} err="failed to get container status \"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633\": rpc error: code = NotFound desc = could not find container \"c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633\": container with ID starting with c5f43b2d718e6d1f99e822b2abc8deded3855547d2b2e2409bb2a4d54dd24633 not found: ID does not exist" Jul 06 19:41:42.529251 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.529227 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:42.582695 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:42.582666 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-f5pqh"] Jul 06 19:41:43.222714 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:43.222681 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977455f8-295c-47eb-9598-28166a297d70" path="/var/lib/kubelet/pods/977455f8-295c-47eb-9598-28166a297d70/volumes" Jul 06 19:41:44.508966 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.508917 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf"] Jul 06 19:41:44.509341 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.509260 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977455f8-295c-47eb-9598-28166a297d70" containerName="manager" Jul 06 19:41:44.509341 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.509271 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="977455f8-295c-47eb-9598-28166a297d70" containerName="manager" Jul 06 19:41:44.509341 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.509336 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="977455f8-295c-47eb-9598-28166a297d70" containerName="manager" Jul 06 19:41:44.513827 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.513811 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.515881 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.515863 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6nfjj\"" Jul 06 19:41:44.525034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.525010 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf"] Jul 06 19:41:44.538062 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.538039 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:44.692617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692583 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692619 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692639 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692678 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692721 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692744 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692765 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692784 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.692830 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.692802 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgb68\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-kube-api-access-pgb68\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794044 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.793951 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794044 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794000 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794109 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794143 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794168 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794196 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794448 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794317 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794448 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794351 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794448 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794361 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794611 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794464 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794611 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794507 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgb68\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-kube-api-access-pgb68\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794611 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794562 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.794763 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.794666 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.795043 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.795013 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.796446 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.796425 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.796627 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.796603 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.803009 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.802988 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.803566 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.803545 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgb68\" (UniqueName: \"kubernetes.io/projected/9b11431b-349f-47d6-9a72-bbe310b8d5b3-kube-api-access-pgb68\") pod \"maas-default-gateway-openshift-default-56654dcf7-gv6vf\" (UID: \"9b11431b-349f-47d6-9a72-bbe310b8d5b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.823991 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.823970 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:44.968837 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.968814 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf"] Jul 06 19:41:44.970788 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:41:44.970759 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b11431b_349f_47d6_9a72_bbe310b8d5b3.slice/crio-eda1f99d41103d77886a08f14a9221c3932941a8a88cdcd86c94bbb6a20a2a88 WatchSource:0}: Error finding container eda1f99d41103d77886a08f14a9221c3932941a8a88cdcd86c94bbb6a20a2a88: Status 404 returned error can't find the container with id eda1f99d41103d77886a08f14a9221c3932941a8a88cdcd86c94bbb6a20a2a88 Jul 06 19:41:44.972898 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.972871 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:41:44.973002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.972948 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:41:44.973002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.972983 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:41:44.975334 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:44.975306 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:45.109811 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.109783 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:45.229799 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.229738 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:45.515046 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.515014 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" event={"ID":"9b11431b-349f-47d6-9a72-bbe310b8d5b3","Type":"ContainerStarted","Data":"38eda01e37bff75785c1cfbebe3466e06f4e806fe525f852323f768f847b4dc3"} Jul 06 19:41:45.515480 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.515065 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" event={"ID":"9b11431b-349f-47d6-9a72-bbe310b8d5b3","Type":"ContainerStarted","Data":"eda1f99d41103d77886a08f14a9221c3932941a8a88cdcd86c94bbb6a20a2a88"} Jul 06 19:41:45.540895 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.540843 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" podStartSLOduration=1.540828108 podStartE2EDuration="1.540828108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:45.539071751 +0000 UTC m=+540.886714926" watchObservedRunningTime="2026-07-06 19:41:45.540828108 +0000 UTC m=+540.888471282" Jul 06 19:41:45.824560 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:45.824486 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:46.829225 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:46.829173 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:47.525024 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:47.524993 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:47.526265 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:47.526240 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:41:49.224155 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:49.224127 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:49.289740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:49.289703 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:50.513707 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:50.513654 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:50.582973 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:50.582920 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:58.439208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:41:58.439173 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpjb9" Jul 06 19:42:00.717077 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.717039 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:00.743310 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.743271 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:00.748431 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.748403 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.751354 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.751327 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7887x\"" Jul 06 19:42:00.751495 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.751470 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:42:00.768115 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.768086 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:00.835241 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.835186 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.835445 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.835369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbvk\" (UniqueName: \"kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.837480 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.837453 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:00.855791 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.855761 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:00.919019 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.918980 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:00.936318 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.936283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbvk\" (UniqueName: \"kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.936520 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.936375 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.937062 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.937041 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:00.946568 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:00.946540 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbvk\" (UniqueName: \"kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk\") pod \"limitador-limitador-64c94fd64f-8smcm\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:01.062128 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:01.062038 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:01.193572 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:01.193534 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:01.196543 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:42:01.196517 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17441dca_dd20_42a1_98fc_eb3e64dd8d3b.slice/crio-5019a973cca1747ee0fd0862602e8571c84a591667afe30edd65f472883f3585 WatchSource:0}: Error finding container 5019a973cca1747ee0fd0862602e8571c84a591667afe30edd65f472883f3585: Status 404 returned error can't find the container with id 5019a973cca1747ee0fd0862602e8571c84a591667afe30edd65f472883f3585 Jul 06 19:42:01.587890 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:01.587852 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" event={"ID":"17441dca-dd20-42a1-98fc-eb3e64dd8d3b","Type":"ContainerStarted","Data":"5019a973cca1747ee0fd0862602e8571c84a591667afe30edd65f472883f3585"} Jul 06 19:42:01.775172 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:01.775135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:01.846599 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:01.846499 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:04.603210 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:04.603178 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" event={"ID":"17441dca-dd20-42a1-98fc-eb3e64dd8d3b","Type":"ContainerStarted","Data":"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28"} Jul 06 19:42:04.603692 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:04.603258 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:04.622386 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:04.622332 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" podStartSLOduration=1.96972167 podStartE2EDuration="4.622317092s" totalImagesPullingTime="2.652595422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:04.621227307 +0000 UTC m=+559.968870484" watchObservedRunningTime="2026-07-06 19:42:04.622317092 +0000 UTC m=+559.969960266" Jul 06 19:42:15.611755 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:15.611727 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:15.954523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:15.954489 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:15.954697 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:15.954672 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" podUID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" containerName="limitador" containerID="cri-o://4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28" gracePeriod=30 Jul 06 19:42:16.501049 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.501024 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:16.580736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.580650 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxbvk\" (UniqueName: \"kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk\") pod \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " Jul 06 19:42:16.580736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.580709 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file\") pod \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\" (UID: \"17441dca-dd20-42a1-98fc-eb3e64dd8d3b\") " Jul 06 19:42:16.581134 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.581105 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file" (OuterVolumeSpecName: "config-file") pod "17441dca-dd20-42a1-98fc-eb3e64dd8d3b" (UID: "17441dca-dd20-42a1-98fc-eb3e64dd8d3b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:42:16.582712 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.582688 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk" (OuterVolumeSpecName: "kube-api-access-fxbvk") pod "17441dca-dd20-42a1-98fc-eb3e64dd8d3b" (UID: "17441dca-dd20-42a1-98fc-eb3e64dd8d3b"). InnerVolumeSpecName "kube-api-access-fxbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:42:16.658787 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.658752 2584 generic.go:356] "Generic (PLEG): container finished" podID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" containerID="4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28" exitCode=0 Jul 06 19:42:16.659291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.658799 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" Jul 06 19:42:16.659291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.658839 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" event={"ID":"17441dca-dd20-42a1-98fc-eb3e64dd8d3b","Type":"ContainerDied","Data":"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28"} Jul 06 19:42:16.659291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.658893 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8smcm" event={"ID":"17441dca-dd20-42a1-98fc-eb3e64dd8d3b","Type":"ContainerDied","Data":"5019a973cca1747ee0fd0862602e8571c84a591667afe30edd65f472883f3585"} Jul 06 19:42:16.659291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.658915 2584 scope.go:117] "RemoveContainer" containerID="4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28" Jul 06 19:42:16.667243 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.667215 2584 scope.go:117] "RemoveContainer" containerID="4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28" Jul 06 19:42:16.667492 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:42:16.667473 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28\": container with ID starting with 4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28 not found: ID does not exist" containerID="4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28" Jul 06 19:42:16.667545 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.667501 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28"} err="failed to get container status \"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28\": rpc error: code = NotFound desc = could not find container \"4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28\": container with ID starting with 4345f21fea10f80f31933a92215e73fd696bc39b56708fa6dbcb5dd7e57adc28 not found: ID does not exist" Jul 06 19:42:16.682197 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.682176 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbvk\" (UniqueName: \"kubernetes.io/projected/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-kube-api-access-fxbvk\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:42:16.682197 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.682196 2584 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17441dca-dd20-42a1-98fc-eb3e64dd8d3b-config-file\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:42:16.693479 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.693453 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:16.694824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:16.694806 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8smcm"] Jul 06 19:42:17.223278 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:17.223246 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" path="/var/lib/kubelet/pods/17441dca-dd20-42a1-98fc-eb3e64dd8d3b/volumes" Jul 06 19:42:18.494548 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.494517 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-64rbd"] Jul 06 19:42:18.494994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.494842 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" containerName="limitador" Jul 06 19:42:18.494994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.494853 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" containerName="limitador" Jul 06 19:42:18.494994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.494917 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="17441dca-dd20-42a1-98fc-eb3e64dd8d3b" containerName="limitador" Jul 06 19:42:18.499470 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.499453 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.501517 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.501493 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:42:18.501633 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.501518 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9xxwp\"" Jul 06 19:42:18.515660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.515630 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-64rbd"] Jul 06 19:42:18.558488 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.558457 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:18.599337 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.599310 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25c99692-2841-4930-afa1-e8235f105604-data\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.599489 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.599340 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wj7z\" (UniqueName: \"kubernetes.io/projected/25c99692-2841-4930-afa1-e8235f105604-kube-api-access-6wj7z\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.700553 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.700515 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25c99692-2841-4930-afa1-e8235f105604-data\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.700553 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.700556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wj7z\" (UniqueName: \"kubernetes.io/projected/25c99692-2841-4930-afa1-e8235f105604-kube-api-access-6wj7z\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.700891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.700874 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25c99692-2841-4930-afa1-e8235f105604-data\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.711084 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.711060 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wj7z\" (UniqueName: \"kubernetes.io/projected/25c99692-2841-4930-afa1-e8235f105604-kube-api-access-6wj7z\") pod \"postgres-69fb4594cf-64rbd\" (UID: \"25c99692-2841-4930-afa1-e8235f105604\") " pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.810284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.810178 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:18.940282 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:18.940074 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-64rbd"] Jul 06 19:42:18.943043 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:42:18.943015 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c99692_2841_4930_afa1_e8235f105604.slice/crio-6d42df66fa145d491661daa406c7a4c823316a0631ca48f4579858aebbeab122 WatchSource:0}: Error finding container 6d42df66fa145d491661daa406c7a4c823316a0631ca48f4579858aebbeab122: Status 404 returned error can't find the container with id 6d42df66fa145d491661daa406c7a4c823316a0631ca48f4579858aebbeab122 Jul 06 19:42:19.675747 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:19.675709 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-64rbd" event={"ID":"25c99692-2841-4930-afa1-e8235f105604","Type":"ContainerStarted","Data":"6d42df66fa145d491661daa406c7a4c823316a0631ca48f4579858aebbeab122"} Jul 06 19:42:23.452524 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:23.452490 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:23.556515 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:23.556486 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:23.698053 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:23.698022 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-64rbd" event={"ID":"25c99692-2841-4930-afa1-e8235f105604","Type":"ContainerStarted","Data":"ed1ebd9783ef3a2f35b7d349e6ccc5d673851f886321ad6df5d194b7a723e1e3"} Jul 06 19:42:23.698284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:23.698264 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:23.718083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:23.717980 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-64rbd" podStartSLOduration=1.164061757 podStartE2EDuration="5.717961449s" totalImagesPullingTime="4.553899692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:23.716083207 +0000 UTC m=+579.063726382" watchObservedRunningTime="2026-07-06 19:42:23.717961449 +0000 UTC m=+579.065604616" Jul 06 19:42:29.732984 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:29.732954 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-64rbd" Jul 06 19:42:30.097991 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:30.097886 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:30.446260 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:30.446226 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:37.120226 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.120191 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:37.139418 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.139387 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:37.157017 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.156991 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.159144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.159122 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:42:37.159291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.159188 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:42:37.159291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.159216 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8h9jv\"" Jul 06 19:42:37.163711 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.163645 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:37.218004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.217964 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:37.275409 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.275373 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhc6\" (UniqueName: \"kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.275561 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.275464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.322279 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.322242 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:37.332651 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.332622 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:42:37.334806 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.334791 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.348636 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.348602 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:42:37.375969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.375881 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhc6\" (UniqueName: \"kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.376110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.375975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.378473 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.378450 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.385354 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.385328 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhc6\" (UniqueName: \"kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6\") pod \"maas-controller-b8f9c595d-gwbc4\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.418628 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.418600 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:37.468967 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.468920 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:37.477700 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.477668 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.477821 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.477797 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bwr\" (UniqueName: \"kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.578652 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.578604 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bwr\" (UniqueName: \"kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.578810 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.578727 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.581388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.581363 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.597458 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.597430 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bwr\" (UniqueName: \"kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr\") pod \"maas-controller-7867bf745b-cmww8\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.621387 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.621367 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:37.623741 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:42:37.623709 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c0388b_920b_4eea_b5c9_da22a6e1cdc1.slice/crio-2d6ee0847f0076b00396300fa116faa3581e6f3a8841fc90b7dd684f39468fb1 WatchSource:0}: Error finding container 2d6ee0847f0076b00396300fa116faa3581e6f3a8841fc90b7dd684f39468fb1: Status 404 returned error can't find the container with id 2d6ee0847f0076b00396300fa116faa3581e6f3a8841fc90b7dd684f39468fb1 Jul 06 19:42:37.644664 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.644638 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:37.761829 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.761798 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" event={"ID":"03c0388b-920b-4eea-b5c9-da22a6e1cdc1","Type":"ContainerStarted","Data":"2d6ee0847f0076b00396300fa116faa3581e6f3a8841fc90b7dd684f39468fb1"} Jul 06 19:42:37.780359 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:37.780210 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:42:37.782744 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:42:37.782723 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bfff1a_85f0_4f81_9dfa_d33a5cbb0742.slice/crio-4e029f93f4bab75e91f4652433ffa99b9a72927c33b9c4738deb019abb534208 WatchSource:0}: Error finding container 4e029f93f4bab75e91f4652433ffa99b9a72927c33b9c4738deb019abb534208: Status 404 returned error can't find the container with id 4e029f93f4bab75e91f4652433ffa99b9a72927c33b9c4738deb019abb534208 Jul 06 19:42:38.770735 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:38.770650 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-cmww8" event={"ID":"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742","Type":"ContainerStarted","Data":"4e029f93f4bab75e91f4652433ffa99b9a72927c33b9c4738deb019abb534208"} Jul 06 19:42:40.783782 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.783724 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" event={"ID":"03c0388b-920b-4eea-b5c9-da22a6e1cdc1","Type":"ContainerStarted","Data":"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b"} Jul 06 19:42:40.784256 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.783803 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:40.785464 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.785438 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-cmww8" event={"ID":"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742","Type":"ContainerStarted","Data":"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087"} Jul 06 19:42:40.785583 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.785482 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:40.819027 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.818838 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" podStartSLOduration=0.833047997 podStartE2EDuration="3.818818288s" totalImagesPullingTime="2.985770291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:40.818026716 +0000 UTC m=+596.165669891" watchObservedRunningTime="2026-07-06 19:42:40.818818288 +0000 UTC m=+596.166461466" Jul 06 19:42:40.866838 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.866739 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7867bf745b-cmww8" podStartSLOduration=1.050441091 podStartE2EDuration="3.86672309s" totalImagesPullingTime="2.816281999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:40.865003153 +0000 UTC m=+596.212646330" watchObservedRunningTime="2026-07-06 19:42:40.86672309 +0000 UTC m=+596.214366264" Jul 06 19:42:40.925139 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:40.925098 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:41.020099 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:41.020069 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:41.121058 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:41.120980 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:51.799819 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:51.799775 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:42:51.802324 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:51.800133 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:51.874891 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:51.874858 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:51.875118 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:51.875089 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" podUID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" containerName="manager" containerID="cri-o://0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b" gracePeriod=30 Jul 06 19:42:52.121962 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.121940 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:52.214864 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.214830 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dhc6\" (UniqueName: \"kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6\") pod \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " Jul 06 19:42:52.215081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.214961 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert\") pod \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\" (UID: \"03c0388b-920b-4eea-b5c9-da22a6e1cdc1\") " Jul 06 19:42:52.216969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.216943 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "03c0388b-920b-4eea-b5c9-da22a6e1cdc1" (UID: "03c0388b-920b-4eea-b5c9-da22a6e1cdc1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:42:52.217080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.217000 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6" (OuterVolumeSpecName: "kube-api-access-6dhc6") pod "03c0388b-920b-4eea-b5c9-da22a6e1cdc1" (UID: "03c0388b-920b-4eea-b5c9-da22a6e1cdc1"). InnerVolumeSpecName "kube-api-access-6dhc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:42:52.316650 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.316612 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhc6\" (UniqueName: \"kubernetes.io/projected/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-kube-api-access-6dhc6\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:42:52.316650 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.316643 2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c0388b-920b-4eea-b5c9-da22a6e1cdc1-webhook-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:42:52.842229 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.842191 2584 generic.go:356] "Generic (PLEG): container finished" podID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" containerID="0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b" exitCode=0 Jul 06 19:42:52.842703 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.842274 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" Jul 06 19:42:52.842703 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.842270 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" event={"ID":"03c0388b-920b-4eea-b5c9-da22a6e1cdc1","Type":"ContainerDied","Data":"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b"} Jul 06 19:42:52.842703 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.842427 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b8f9c595d-gwbc4" event={"ID":"03c0388b-920b-4eea-b5c9-da22a6e1cdc1","Type":"ContainerDied","Data":"2d6ee0847f0076b00396300fa116faa3581e6f3a8841fc90b7dd684f39468fb1"} Jul 06 19:42:52.842703 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.842453 2584 scope.go:117] "RemoveContainer" containerID="0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b" Jul 06 19:42:52.850516 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.850500 2584 scope.go:117] "RemoveContainer" containerID="0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b" Jul 06 19:42:52.850789 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:42:52.850769 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b\": container with ID starting with 0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b not found: ID does not exist" containerID="0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b" Jul 06 19:42:52.850853 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.850803 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b"} err="failed to get container status \"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b\": rpc error: code = NotFound desc = could not find container \"0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b\": container with ID starting with 0e654c34bf1e08a74330be558993ba7c33da41f6722b4af54c67382ce93bc07b not found: ID does not exist" Jul 06 19:42:52.865402 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.865376 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:52.873526 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:52.873504 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b8f9c595d-gwbc4"] Jul 06 19:42:53.222555 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:42:53.222527 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" path="/var/lib/kubelet/pods/03c0388b-920b-4eea-b5c9-da22a6e1cdc1/volumes" Jul 06 19:43:13.365511 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.365463 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:13.932737 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.932702 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-554c6857df-kg4lf"] Jul 06 19:43:13.933084 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.933071 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" containerName="manager" Jul 06 19:43:13.933129 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.933085 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" containerName="manager" Jul 06 19:43:13.933167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.933134 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c0388b-920b-4eea-b5c9-da22a6e1cdc1" containerName="manager" Jul 06 19:43:13.985158 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.985123 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-554c6857df-kg4lf"] Jul 06 19:43:13.985304 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.985251 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:13.987653 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.987623 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 19:43:13.988270 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:13.988256 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-2fvtb\"" Jul 06 19:43:14.112255 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.112218 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjk9\" (UniqueName: \"kubernetes.io/projected/11b52f75-be8b-4508-86dd-5e53e559aac2-kube-api-access-6rjk9\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.112453 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.112281 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b52f75-be8b-4508-86dd-5e53e559aac2-openshift-service-ca\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.112453 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.112337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b52f75-be8b-4508-86dd-5e53e559aac2-maas-api-tls\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.213641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.213544 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjk9\" (UniqueName: \"kubernetes.io/projected/11b52f75-be8b-4508-86dd-5e53e559aac2-kube-api-access-6rjk9\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.213641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.213608 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b52f75-be8b-4508-86dd-5e53e559aac2-openshift-service-ca\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.213641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.213633 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b52f75-be8b-4508-86dd-5e53e559aac2-maas-api-tls\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.214314 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.214293 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11b52f75-be8b-4508-86dd-5e53e559aac2-openshift-service-ca\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.216059 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.216039 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11b52f75-be8b-4508-86dd-5e53e559aac2-maas-api-tls\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.222144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.222119 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjk9\" (UniqueName: \"kubernetes.io/projected/11b52f75-be8b-4508-86dd-5e53e559aac2-kube-api-access-6rjk9\") pod \"maas-api-554c6857df-kg4lf\" (UID: \"11b52f75-be8b-4508-86dd-5e53e559aac2\") " pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.295474 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.295440 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:14.442531 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.442507 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-554c6857df-kg4lf"] Jul 06 19:43:14.445055 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:43:14.445015 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b52f75_be8b_4508_86dd_5e53e559aac2.slice/crio-1c7ecb65b254d7409fad0cd5996de41352e212e41753b54cc52312e4d6484faf WatchSource:0}: Error finding container 1c7ecb65b254d7409fad0cd5996de41352e212e41753b54cc52312e4d6484faf: Status 404 returned error can't find the container with id 1c7ecb65b254d7409fad0cd5996de41352e212e41753b54cc52312e4d6484faf Jul 06 19:43:14.504333 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.504260 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pmlkj"] Jul 06 19:43:14.507378 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.507362 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.509567 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.509546 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:43:14.517745 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.517555 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pmlkj"] Jul 06 19:43:14.517745 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.517617 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b58\" (UniqueName: \"kubernetes.io/projected/8fd2b566-4b66-4728-a953-df5b2acb65b8-kube-api-access-w7b58\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.517745 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.517659 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd2b566-4b66-4728-a953-df5b2acb65b8-plugins-config-volume\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.567754 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.567714 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mw7np"] Jul 06 19:43:14.570128 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.570100 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.572495 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.572476 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7wm6x\"" Jul 06 19:43:14.581834 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.581810 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mw7np"] Jul 06 19:43:14.618583 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.618548 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b58\" (UniqueName: \"kubernetes.io/projected/8fd2b566-4b66-4728-a953-df5b2acb65b8-kube-api-access-w7b58\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.618750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.618600 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd2b566-4b66-4728-a953-df5b2acb65b8-plugins-config-volume\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.618750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.618634 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlb85\" (UniqueName: \"kubernetes.io/projected/4ededa9f-2440-478e-8ec5-4bf9292c2151-kube-api-access-wlb85\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.618750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.618687 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ededa9f-2440-478e-8ec5-4bf9292c2151-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.619306 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.619289 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd2b566-4b66-4728-a953-df5b2acb65b8-plugins-config-volume\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.626344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.626320 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b58\" (UniqueName: \"kubernetes.io/projected/8fd2b566-4b66-4728-a953-df5b2acb65b8-kube-api-access-w7b58\") pod \"payload-processing-58f86c7798-pmlkj\" (UID: \"8fd2b566-4b66-4728-a953-df5b2acb65b8\") " pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.720079 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.720046 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlb85\" (UniqueName: \"kubernetes.io/projected/4ededa9f-2440-478e-8ec5-4bf9292c2151-kube-api-access-wlb85\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.720221 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.720108 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ededa9f-2440-478e-8ec5-4bf9292c2151-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.720847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.720821 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ededa9f-2440-478e-8ec5-4bf9292c2151-plugins-config-volume\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.730032 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.730011 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlb85\" (UniqueName: \"kubernetes.io/projected/4ededa9f-2440-478e-8ec5-4bf9292c2151-kube-api-access-wlb85\") pod \"payload-pre-processing-7964457696-mw7np\" (UID: \"4ededa9f-2440-478e-8ec5-4bf9292c2151\") " pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.820131 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.820041 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:14.882360 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.882297 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:14.944769 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.944735 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-554c6857df-kg4lf" event={"ID":"11b52f75-be8b-4508-86dd-5e53e559aac2","Type":"ContainerStarted","Data":"1c7ecb65b254d7409fad0cd5996de41352e212e41753b54cc52312e4d6484faf"} Jul 06 19:43:14.956256 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.956151 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pmlkj"] Jul 06 19:43:14.959245 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:43:14.959218 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd2b566_4b66_4728_a953_df5b2acb65b8.slice/crio-077af6a73039642a94f82b03799ab354295055fa03b32a6438b07390ec27acc0 WatchSource:0}: Error finding container 077af6a73039642a94f82b03799ab354295055fa03b32a6438b07390ec27acc0: Status 404 returned error can't find the container with id 077af6a73039642a94f82b03799ab354295055fa03b32a6438b07390ec27acc0 Jul 06 19:43:14.995213 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:14.994953 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:15.027430 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:15.027409 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-mw7np"] Jul 06 19:43:15.030101 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:43:15.030069 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ededa9f_2440_478e_8ec5_4bf9292c2151.slice/crio-3b29374c9f2cf36c6107c44f23209c7f65530f070284a7e872de7578648f5ca3 WatchSource:0}: Error finding container 3b29374c9f2cf36c6107c44f23209c7f65530f070284a7e872de7578648f5ca3: Status 404 returned error can't find the container with id 3b29374c9f2cf36c6107c44f23209c7f65530f070284a7e872de7578648f5ca3 Jul 06 19:43:15.729998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:15.729960 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:15.954025 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:15.953987 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" event={"ID":"8fd2b566-4b66-4728-a953-df5b2acb65b8","Type":"ContainerStarted","Data":"077af6a73039642a94f82b03799ab354295055fa03b32a6438b07390ec27acc0"} Jul 06 19:43:15.956500 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:15.956450 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" event={"ID":"4ededa9f-2440-478e-8ec5-4bf9292c2151","Type":"ContainerStarted","Data":"3b29374c9f2cf36c6107c44f23209c7f65530f070284a7e872de7578648f5ca3"} Jul 06 19:43:17.993509 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.993470 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-554c6857df-kg4lf" event={"ID":"11b52f75-be8b-4508-86dd-5e53e559aac2","Type":"ContainerStarted","Data":"5ebee53b1789c7267b3937dcd63a87d5c72cea678491046983ec4edfe9023f5b"} Jul 06 19:43:17.994051 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.994027 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:17.995639 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.995619 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" event={"ID":"8fd2b566-4b66-4728-a953-df5b2acb65b8","Type":"ContainerStarted","Data":"af3b9bfd8cda912e5355f266c92ad71eb768c975b7e5fc20f291d34934c58727"} Jul 06 19:43:17.995782 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.995757 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:17.997257 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.997236 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" event={"ID":"4ededa9f-2440-478e-8ec5-4bf9292c2151","Type":"ContainerStarted","Data":"3e1dce1af11a1520514d0c5ff1a156c0851e3ab47c4e2d825f7ab04dd19caff9"} Jul 06 19:43:17.997366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:17.997353 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:18.014638 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:18.014592 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-554c6857df-kg4lf" podStartSLOduration=2.538993099 podStartE2EDuration="5.014577797s" totalImagesPullingTime="2.475584698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:18.012439174 +0000 UTC m=+633.360082350" watchObservedRunningTime="2026-07-06 19:43:18.014577797 +0000 UTC m=+633.362220971" Jul 06 19:43:18.036685 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:18.036643 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" podStartSLOduration=2.07610845 podStartE2EDuration="4.036629645s" totalImagesPullingTime="1.960521195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:18.035012551 +0000 UTC m=+633.382655726" watchObservedRunningTime="2026-07-06 19:43:18.036629645 +0000 UTC m=+633.384272821" Jul 06 19:43:18.050026 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:18.049975 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" podStartSLOduration=2.157964731 podStartE2EDuration="4.049957799s" totalImagesPullingTime="1.891993068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:18.049408338 +0000 UTC m=+633.397051512" watchObservedRunningTime="2026-07-06 19:43:18.049957799 +0000 UTC m=+633.397600968" Jul 06 19:43:20.731546 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:20.731507 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:20.835048 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:20.835010 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:23.023607 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.023574 2584 generic.go:356] "Generic (PLEG): container finished" podID="9b11431b-349f-47d6-9a72-bbe310b8d5b3" containerID="38eda01e37bff75785c1cfbebe3466e06f4e806fe525f852323f768f847b4dc3" exitCode=0 Jul 06 19:43:23.024041 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.023631 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" event={"ID":"9b11431b-349f-47d6-9a72-bbe310b8d5b3","Type":"ContainerDied","Data":"38eda01e37bff75785c1cfbebe3466e06f4e806fe525f852323f768f847b4dc3"} Jul 06 19:43:23.024041 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.024022 2584 scope.go:117] "RemoveContainer" containerID="38eda01e37bff75785c1cfbebe3466e06f4e806fe525f852323f768f847b4dc3" Jul 06 19:43:23.025002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.024974 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:43:23.025109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.025059 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:43:23.025171 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.025106 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:43:23.989785 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:23.989496 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:24.006963 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.006921 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-554c6857df-kg4lf" Jul 06 19:43:24.030646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.030617 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" event={"ID":"9b11431b-349f-47d6-9a72-bbe310b8d5b3","Type":"ContainerStarted","Data":"5e97427b3d853f932653bdf755096462cdbaf7ceeb951c6731c345995aaaf11e"} Jul 06 19:43:24.083958 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.075857 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:24.824947 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.824894 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:43:24.825167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.825143 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gv6vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 19:43:24.825222 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:24.825192 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" podUID="9b11431b-349f-47d6-9a72-bbe310b8d5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 19:43:25.029117 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:25.029081 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:25.128792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:25.128697 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:25.824779 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:25.824742 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gv6vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 19:43:25.824980 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:25.824798 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" podUID="9b11431b-349f-47d6-9a72-bbe310b8d5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 19:43:26.824945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:26.824888 2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gv6vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 19:43:26.825358 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:26.824971 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" podUID="9b11431b-349f-47d6-9a72-bbe310b8d5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 19:43:27.846940 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:27.846897 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:43:27.847395 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:27.847195 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:43:27.848050 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:27.848033 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gv6vf" Jul 06 19:43:29.003719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:29.003691 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-mw7np" Jul 06 19:43:29.004228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:29.003764 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-pmlkj" Jul 06 19:43:43.305490 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.305453 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km"] Jul 06 19:43:43.310498 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.310460 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.313727 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.313707 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g78mn\"" Jul 06 19:43:43.314435 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.314413 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:43:43.314603 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.314579 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:43:43.314721 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.314607 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 19:43:43.324795 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.324772 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km"] Jul 06 19:43:43.327779 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.327755 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:43.391523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391487 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.391523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391520 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2116eb-23f4-4d75-8127-7a8750bd4772-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.391773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391561 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.391773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391653 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.391773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391732 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skj79\" (UniqueName: \"kubernetes.io/projected/5b2116eb-23f4-4d75-8127-7a8750bd4772-kube-api-access-skj79\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.391922 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.391849 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492569 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492536 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492569 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492572 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492589 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2116eb-23f4-4d75-8127-7a8750bd4772-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492628 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.492824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.492702 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skj79\" (UniqueName: \"kubernetes.io/projected/5b2116eb-23f4-4d75-8127-7a8750bd4772-kube-api-access-skj79\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.493080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.493054 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.493125 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.493095 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.493160 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.493137 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.494834 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.494813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b2116eb-23f4-4d75-8127-7a8750bd4772-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.495059 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.495040 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2116eb-23f4-4d75-8127-7a8750bd4772-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.501113 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.501095 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skj79\" (UniqueName: \"kubernetes.io/projected/5b2116eb-23f4-4d75-8127-7a8750bd4772-kube-api-access-skj79\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km\" (UID: \"5b2116eb-23f4-4d75-8127-7a8750bd4772\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.622719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.622636 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:43.770460 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.770436 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km"] Jul 06 19:43:43.773114 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:43:43.773085 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2116eb_23f4_4d75_8127_7a8750bd4772.slice/crio-7d77fea971f8d84b8af960559daf3c481841bb935d12cc1820d197f6f3e2c79f WatchSource:0}: Error finding container 7d77fea971f8d84b8af960559daf3c481841bb935d12cc1820d197f6f3e2c79f: Status 404 returned error can't find the container with id 7d77fea971f8d84b8af960559daf3c481841bb935d12cc1820d197f6f3e2c79f Jul 06 19:43:43.774560 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:43.774543 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:43:44.131750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:44.131709 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" event={"ID":"5b2116eb-23f4-4d75-8127-7a8750bd4772","Type":"ContainerStarted","Data":"7d77fea971f8d84b8af960559daf3c481841bb935d12cc1820d197f6f3e2c79f"} Jul 06 19:43:45.137606 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:45.137577 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" event={"ID":"5b2116eb-23f4-4d75-8127-7a8750bd4772","Type":"ContainerStarted","Data":"5bc5608f2482b5546089f424113c6ae211f877d2c8269ecd1b58134e1e7c745a"} Jul 06 19:43:45.158766 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:45.158558 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" podStartSLOduration=1.018213545 podStartE2EDuration="2.158538997s" totalImagesPullingTime="1.140325452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:45.158176173 +0000 UTC m=+660.505819383" watchObservedRunningTime="2026-07-06 19:43:45.158538997 +0000 UTC m=+660.506182173" Jul 06 19:43:53.623283 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:53.623238 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:53.623759 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:53.623318 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:53.635870 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:53.635843 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:54.190016 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:54.189989 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km" Jul 06 19:43:55.204329 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:43:55.204292 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:12.304292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:12.304253 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:15.576076 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:15.576036 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:15.652287 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:15.652253 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:16.632199 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:16.632168 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:20.574829 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.574789 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws"] Jul 06 19:44:20.581599 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.581576 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.583643 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.583622 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:44:20.589975 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.589948 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws"] Jul 06 19:44:20.618282 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.618248 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:20.723590 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723560 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.723769 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723607 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.723769 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723628 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db77586-64c4-4215-bbd4-c7b52fcaab84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.723769 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723702 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj4t\" (UniqueName: \"kubernetes.io/projected/7db77586-64c4-4215-bbd4-c7b52fcaab84-kube-api-access-drj4t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.723769 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723734 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.723916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.723797 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824441 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824459 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824488 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db77586-64c4-4215-bbd4-c7b52fcaab84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj4t\" (UniqueName: \"kubernetes.io/projected/7db77586-64c4-4215-bbd4-c7b52fcaab84-kube-api-access-drj4t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824632 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824558 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824970 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824866 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.824970 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824960 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.825080 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.824999 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.826776 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.826745 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db77586-64c4-4215-bbd4-c7b52fcaab84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.827208 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.827189 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db77586-64c4-4215-bbd4-c7b52fcaab84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.834675 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.834653 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj4t\" (UniqueName: \"kubernetes.io/projected/7db77586-64c4-4215-bbd4-c7b52fcaab84-kube-api-access-drj4t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws\" (UID: \"7db77586-64c4-4215-bbd4-c7b52fcaab84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:20.892328 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:20.892295 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:21.027914 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:21.027879 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws"] Jul 06 19:44:21.031079 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:44:21.031044 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db77586_64c4_4215_bbd4_c7b52fcaab84.slice/crio-0257f041e370bf5f8920f3f378212a9ffd2777cb09ed56e4d014f2794e76b07c WatchSource:0}: Error finding container 0257f041e370bf5f8920f3f378212a9ffd2777cb09ed56e4d014f2794e76b07c: Status 404 returned error can't find the container with id 0257f041e370bf5f8920f3f378212a9ffd2777cb09ed56e4d014f2794e76b07c Jul 06 19:44:21.304999 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:21.304967 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" event={"ID":"7db77586-64c4-4215-bbd4-c7b52fcaab84","Type":"ContainerStarted","Data":"37702ea552aa6c603b398844fabab06fe8c242616d429fe607ae17e869f49344"} Jul 06 19:44:21.305187 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:21.305031 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" event={"ID":"7db77586-64c4-4215-bbd4-c7b52fcaab84","Type":"ContainerStarted","Data":"0257f041e370bf5f8920f3f378212a9ffd2777cb09ed56e4d014f2794e76b07c"} Jul 06 19:44:21.321999 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:21.321946 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" podStartSLOduration=1.134305827 podStartE2EDuration="1.321910087s" totalImagesPullingTime="187.60426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:21.321006011 +0000 UTC m=+696.668649182" watchObservedRunningTime="2026-07-06 19:44:21.321910087 +0000 UTC m=+696.669553263" Jul 06 19:44:22.333504 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:22.333475 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:30.398293 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:30.398249 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:30.892752 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:30.892717 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:30.892968 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:30.892824 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:30.905404 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:30.905375 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:44:31.362815 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:44:31.362780 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws" Jul 06 19:45:00.152818 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.152740 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-5sdc8"] Jul 06 19:45:00.156584 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.156564 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:00.164702 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.164664 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-5sdc8"] Jul 06 19:45:00.289209 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.289169 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qhd6\" (UniqueName: \"kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6\") pod \"maas-api-key-cleanup-29722785-5sdc8\" (UID: \"3eb94900-ecce-4cc9-9fd7-69bdf040000d\") " pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:00.390329 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.390295 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qhd6\" (UniqueName: \"kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6\") pod \"maas-api-key-cleanup-29722785-5sdc8\" (UID: \"3eb94900-ecce-4cc9-9fd7-69bdf040000d\") " pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:00.398999 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.398977 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qhd6\" (UniqueName: \"kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6\") pod \"maas-api-key-cleanup-29722785-5sdc8\" (UID: \"3eb94900-ecce-4cc9-9fd7-69bdf040000d\") " pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:00.467847 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.467820 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:00.595269 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:00.595235 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-5sdc8"] Jul 06 19:45:00.599281 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:45:00.599252 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb94900_ecce_4cc9_9fd7_69bdf040000d.slice/crio-af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad WatchSource:0}: Error finding container af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad: Status 404 returned error can't find the container with id af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad Jul 06 19:45:01.495196 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:01.495167 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" event={"ID":"3eb94900-ecce-4cc9-9fd7-69bdf040000d","Type":"ContainerStarted","Data":"af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad"} Jul 06 19:45:17.573147 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:17.573052 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" event={"ID":"3eb94900-ecce-4cc9-9fd7-69bdf040000d","Type":"ContainerStarted","Data":"1aa0cb9753c96423082f89b1e534f85a74aa0bc76d7c8c4deb8257397d7576b6"} Jul 06 19:45:17.589010 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:17.588955 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" podStartSLOduration=1.31512722 podStartE2EDuration="17.588938905s" totalImagesPullingTime="16.273811685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:17.587107855 +0000 UTC m=+752.934751030" watchObservedRunningTime="2026-07-06 19:45:17.588938905 +0000 UTC m=+752.936582070" Jul 06 19:45:18.579022 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:18.578989 2584 generic.go:356] "Generic (PLEG): container finished" podID="3eb94900-ecce-4cc9-9fd7-69bdf040000d" containerID="1aa0cb9753c96423082f89b1e534f85a74aa0bc76d7c8c4deb8257397d7576b6" exitCode=0 Jul 06 19:45:18.579403 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:18.579062 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" event={"ID":"3eb94900-ecce-4cc9-9fd7-69bdf040000d","Type":"ContainerDied","Data":"1aa0cb9753c96423082f89b1e534f85a74aa0bc76d7c8c4deb8257397d7576b6"} Jul 06 19:45:19.714060 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:19.714037 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:19.874419 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:19.874328 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qhd6\" (UniqueName: \"kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6\") pod \"3eb94900-ecce-4cc9-9fd7-69bdf040000d\" (UID: \"3eb94900-ecce-4cc9-9fd7-69bdf040000d\") " Jul 06 19:45:19.876354 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:19.876322 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6" (OuterVolumeSpecName: "kube-api-access-4qhd6") pod "3eb94900-ecce-4cc9-9fd7-69bdf040000d" (UID: "3eb94900-ecce-4cc9-9fd7-69bdf040000d"). InnerVolumeSpecName "kube-api-access-4qhd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:19.975998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:19.975963 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qhd6\" (UniqueName: \"kubernetes.io/projected/3eb94900-ecce-4cc9-9fd7-69bdf040000d-kube-api-access-4qhd6\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:45:20.589363 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:20.589331 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" event={"ID":"3eb94900-ecce-4cc9-9fd7-69bdf040000d","Type":"ContainerDied","Data":"af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad"} Jul 06 19:45:20.589536 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:20.589377 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af21562c70cf3fe87ec954a4d7169598cb8ef36725aade9e742dd789e0f882ad" Jul 06 19:45:20.589536 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:20.589340 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-5sdc8" Jul 06 19:45:27.735086 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:27.735049 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:27.834041 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:27.834006 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:28.924403 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:28.924366 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:29.033259 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:29.033203 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:52.791981 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:52.791910 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7867bf745b-cmww8" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:45:59.770404 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:45:59.770363 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:02.791709 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:02.791654 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7867bf745b-cmww8" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:46:04.265422 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:04.265381 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:58.232009 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.231974 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:58.338519 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.338485 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:46:58.338704 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.338679 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7867bf745b-cmww8" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" containerID="cri-o://7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087" gracePeriod=30 Jul 06 19:46:58.590832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.590807 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:46:58.682916 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.682881 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6bwr\" (UniqueName: \"kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr\") pod \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " Jul 06 19:46:58.683107 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.682951 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert\") pod \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\" (UID: \"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742\") " Jul 06 19:46:58.684880 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.684859 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" (UID: "36bfff1a-85f0-4f81-9dfa-d33a5cbb0742"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:46:58.684994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.684977 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr" (OuterVolumeSpecName: "kube-api-access-t6bwr") pod "36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" (UID: "36bfff1a-85f0-4f81-9dfa-d33a5cbb0742"). InnerVolumeSpecName "kube-api-access-t6bwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:58.784748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.784661 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bwr\" (UniqueName: \"kubernetes.io/projected/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-kube-api-access-t6bwr\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:46:58.784748 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:58.784691 2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742-webhook-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:46:59.067002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.066896 2584 generic.go:356] "Generic (PLEG): container finished" podID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerID="7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087" exitCode=0 Jul 06 19:46:59.067002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.066956 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-cmww8" Jul 06 19:46:59.067002 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.066967 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-cmww8" event={"ID":"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742","Type":"ContainerDied","Data":"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087"} Jul 06 19:46:59.067263 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.067034 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-cmww8" event={"ID":"36bfff1a-85f0-4f81-9dfa-d33a5cbb0742","Type":"ContainerDied","Data":"4e029f93f4bab75e91f4652433ffa99b9a72927c33b9c4738deb019abb534208"} Jul 06 19:46:59.067263 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.067068 2584 scope.go:117] "RemoveContainer" containerID="7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087" Jul 06 19:46:59.081741 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.081717 2584 scope.go:117] "RemoveContainer" containerID="7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087" Jul 06 19:46:59.082068 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:46:59.082047 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087\": container with ID starting with 7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087 not found: ID does not exist" containerID="7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087" Jul 06 19:46:59.082154 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.082081 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087"} err="failed to get container status \"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087\": rpc error: code = NotFound desc = could not find container \"7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087\": container with ID starting with 7c810cd1edc9899f7df8b61c5ca46c1a9fa731eaddd8cb90f280542181866087 not found: ID does not exist" Jul 06 19:46:59.093110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.093084 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:46:59.097151 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.097127 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7867bf745b-cmww8"] Jul 06 19:46:59.223624 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.223592 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" path="/var/lib/kubelet/pods/36bfff1a-85f0-4f81-9dfa-d33a5cbb0742/volumes" Jul 06 19:46:59.761344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.761310 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:59.896734 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.896704 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7867bf745b-6299t"] Jul 06 19:46:59.897101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897087 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb94900-ecce-4cc9-9fd7-69bdf040000d" containerName="cleanup" Jul 06 19:46:59.897101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897101 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb94900-ecce-4cc9-9fd7-69bdf040000d" containerName="cleanup" Jul 06 19:46:59.897189 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897111 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" Jul 06 19:46:59.897189 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897116 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" Jul 06 19:46:59.897189 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897187 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb94900-ecce-4cc9-9fd7-69bdf040000d" containerName="cleanup" Jul 06 19:46:59.897291 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.897197 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36bfff1a-85f0-4f81-9dfa-d33a5cbb0742" containerName="manager" Jul 06 19:46:59.901603 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.901586 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:46:59.903470 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.903442 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:46:59.903948 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.903921 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:46:59.904046 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.904000 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8h9jv\"" Jul 06 19:46:59.908215 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.908194 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867bf745b-6299t"] Jul 06 19:46:59.995238 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.995204 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zzw\" (UniqueName: \"kubernetes.io/projected/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-kube-api-access-k2zzw\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:46:59.995433 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:46:59.995364 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-webhook-cert\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.096646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.096560 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-webhook-cert\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.096646 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.096611 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zzw\" (UniqueName: \"kubernetes.io/projected/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-kube-api-access-k2zzw\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.099028 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.099002 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-webhook-cert\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.105420 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.105395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zzw\" (UniqueName: \"kubernetes.io/projected/62cce9ad-9dbc-43cc-9df4-0bb84d980a68-kube-api-access-k2zzw\") pod \"maas-controller-7867bf745b-6299t\" (UID: \"62cce9ad-9dbc-43cc-9df4-0bb84d980a68\") " pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.213477 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.213428 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:00.348897 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:00.348693 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867bf745b-6299t"] Jul 06 19:47:00.351752 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:47:00.351727 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cce9ad_9dbc_43cc_9df4_0bb84d980a68.slice/crio-8ef6e691d735c691d350206e8af32f477288afa20612cb05af3e811f52808457 WatchSource:0}: Error finding container 8ef6e691d735c691d350206e8af32f477288afa20612cb05af3e811f52808457: Status 404 returned error can't find the container with id 8ef6e691d735c691d350206e8af32f477288afa20612cb05af3e811f52808457 Jul 06 19:47:01.079511 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:01.079478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-6299t" event={"ID":"62cce9ad-9dbc-43cc-9df4-0bb84d980a68","Type":"ContainerStarted","Data":"0b8aeda05738e03676f8ba21121ef47b8ac71f6ba9495efa6cc6f5af9aaf7d04"} Jul 06 19:47:01.079885 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:01.079527 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867bf745b-6299t" event={"ID":"62cce9ad-9dbc-43cc-9df4-0bb84d980a68","Type":"ContainerStarted","Data":"8ef6e691d735c691d350206e8af32f477288afa20612cb05af3e811f52808457"} Jul 06 19:47:01.079885 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:01.079612 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:01.099289 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:01.099194 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7867bf745b-6299t" podStartSLOduration=1.610039835 podStartE2EDuration="2.099177102s" totalImagesPullingTime="489.137267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:01.09753164 +0000 UTC m=+856.445174816" watchObservedRunningTime="2026-07-06 19:47:01.099177102 +0000 UTC m=+856.446820316" Jul 06 19:47:12.092065 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:12.092033 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7867bf745b-6299t" Jul 06 19:47:40.698375 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:40.698293 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:40.799988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:40.799947 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:40.966237 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:40.966062 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:46.863064 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:46.863027 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:46.933344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:46.933309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:47.078737 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:47.078696 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:47.240328 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:47.240294 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:47.337807 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:47.337776 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:53.070508 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:53.070470 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:53.167972 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:47:53.167936 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:23.150106 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:48:23.150067 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:48:24.236145 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:48:24.236110 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:01.660565 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:50:01.660525 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:51.633851 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:50:51.633774 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:51.725892 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:50:51.725858 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:54.038998 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:50:54.038960 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:02.604062 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:51:02.604029 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:02.703641 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:51:02.703599 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:59.096382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:51:59.096343 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:51:59.097048 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:51:59.096641 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" podUID="756a2264-2fd7-40eb-b097-4fee43934078" containerName="manager" containerID="cri-o://8c89a3671dca4980ca95d9722201f32e344297fcabd02d87cba9833c4f949ae1" gracePeriod=10 Jul 06 19:52:01.450156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:01.450117 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" podUID="756a2264-2fd7-40eb-b097-4fee43934078" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Jul 06 19:52:01.559376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:01.559289 2584 generic.go:356] "Generic (PLEG): container finished" podID="756a2264-2fd7-40eb-b097-4fee43934078" containerID="8c89a3671dca4980ca95d9722201f32e344297fcabd02d87cba9833c4f949ae1" exitCode=0 Jul 06 19:52:01.559376 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:01.559367 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" event={"ID":"756a2264-2fd7-40eb-b097-4fee43934078","Type":"ContainerDied","Data":"8c89a3671dca4980ca95d9722201f32e344297fcabd02d87cba9833c4f949ae1"} Jul 06 19:52:01.957108 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:01.957085 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:52:02.047922 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.047889 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume\") pod \"756a2264-2fd7-40eb-b097-4fee43934078\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " Jul 06 19:52:02.048109 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.047994 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blxcg\" (UniqueName: \"kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg\") pod \"756a2264-2fd7-40eb-b097-4fee43934078\" (UID: \"756a2264-2fd7-40eb-b097-4fee43934078\") " Jul 06 19:52:02.048280 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.048257 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "756a2264-2fd7-40eb-b097-4fee43934078" (UID: "756a2264-2fd7-40eb-b097-4fee43934078"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:52:02.050013 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.049991 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg" (OuterVolumeSpecName: "kube-api-access-blxcg") pod "756a2264-2fd7-40eb-b097-4fee43934078" (UID: "756a2264-2fd7-40eb-b097-4fee43934078"). InnerVolumeSpecName "kube-api-access-blxcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:52:02.148872 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.148787 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/756a2264-2fd7-40eb-b097-4fee43934078-extensions-socket-volume\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:52:02.148872 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.148813 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blxcg\" (UniqueName: \"kubernetes.io/projected/756a2264-2fd7-40eb-b097-4fee43934078-kube-api-access-blxcg\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:52:02.565740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.565704 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" event={"ID":"756a2264-2fd7-40eb-b097-4fee43934078","Type":"ContainerDied","Data":"97341a35615ad4c0237b4c1e4fcd81b5a800952d57f8d7c9fba2a6108dcfd95c"} Jul 06 19:52:02.565740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.565734 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh" Jul 06 19:52:02.566308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.565763 2584 scope.go:117] "RemoveContainer" containerID="8c89a3671dca4980ca95d9722201f32e344297fcabd02d87cba9833c4f949ae1" Jul 06 19:52:02.587876 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.587849 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:52:02.591937 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:02.591900 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8f4sh"] Jul 06 19:52:03.222971 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:03.222920 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="756a2264-2fd7-40eb-b097-4fee43934078" path="/var/lib/kubelet/pods/756a2264-2fd7-40eb-b097-4fee43934078/volumes" Jul 06 19:52:07.740144 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:07.740064 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:52.587235 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:52:52.587198 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:00.357755 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.357722 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd"] Jul 06 19:53:00.358226 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.358207 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756a2264-2fd7-40eb-b097-4fee43934078" containerName="manager" Jul 06 19:53:00.358226 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.358228 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="756a2264-2fd7-40eb-b097-4fee43934078" containerName="manager" Jul 06 19:53:00.358348 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.358328 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="756a2264-2fd7-40eb-b097-4fee43934078" containerName="manager" Jul 06 19:53:00.362521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.362504 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.365108 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.365089 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-htwnp\"" Jul 06 19:53:00.377626 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.377597 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd"] Jul 06 19:53:00.486596 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.486556 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/497178e9-ba0b-4410-9f48-69e2b9d04673-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.486789 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.486672 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshsl\" (UniqueName: \"kubernetes.io/projected/497178e9-ba0b-4410-9f48-69e2b9d04673-kube-api-access-lshsl\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.587957 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.587882 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/497178e9-ba0b-4410-9f48-69e2b9d04673-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.588101 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.588010 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshsl\" (UniqueName: \"kubernetes.io/projected/497178e9-ba0b-4410-9f48-69e2b9d04673-kube-api-access-lshsl\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.588344 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.588321 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/497178e9-ba0b-4410-9f48-69e2b9d04673-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.600436 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.600405 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshsl\" (UniqueName: \"kubernetes.io/projected/497178e9-ba0b-4410-9f48-69e2b9d04673-kube-api-access-lshsl\") pod \"kuadrant-operator-controller-manager-86878c769f-8fncd\" (UID: \"497178e9-ba0b-4410-9f48-69e2b9d04673\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.673136 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.673099 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:00.810915 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.810878 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd"] Jul 06 19:53:00.814952 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:53:00.814908 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497178e9_ba0b_4410_9f48_69e2b9d04673.slice/crio-ce910e1274ca8899462473d866517a9f9cfec7188a3a038260a6442ba8fd240a WatchSource:0}: Error finding container ce910e1274ca8899462473d866517a9f9cfec7188a3a038260a6442ba8fd240a: Status 404 returned error can't find the container with id ce910e1274ca8899462473d866517a9f9cfec7188a3a038260a6442ba8fd240a Jul 06 19:53:00.852900 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:00.852871 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" event={"ID":"497178e9-ba0b-4410-9f48-69e2b9d04673","Type":"ContainerStarted","Data":"ce910e1274ca8899462473d866517a9f9cfec7188a3a038260a6442ba8fd240a"} Jul 06 19:53:01.859752 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:01.859716 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" event={"ID":"497178e9-ba0b-4410-9f48-69e2b9d04673","Type":"ContainerStarted","Data":"192aa34d933e0a51c3b507e7c0bac4ef1e65c89ebdc1b6c6b92e1d7afd804c66"} Jul 06 19:53:01.860189 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:01.859784 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:53:01.889986 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:01.889906 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" podStartSLOduration=1.8898908909999999 podStartE2EDuration="1.889890891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:01.889585737 +0000 UTC m=+1217.237228925" watchObservedRunningTime="2026-07-06 19:53:01.889890891 +0000 UTC m=+1217.237534066" Jul 06 19:53:12.867782 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:53:12.867752 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8fncd" Jul 06 19:58:09.868497 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:09.868458 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:21.494921 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:21.494880 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:21.594918 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:21.594880 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:40.847202 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:40.847166 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:46.694308 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:46.694275 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:48.419384 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:48.419352 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:56.691886 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:58:56.691850 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:01.780153 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:01.780119 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:04.642161 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:04.642128 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:04.741913 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:04.741886 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:05.045110 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:05.045070 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:11.086475 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.086433 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:11.163170 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.163135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:11.257717 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.257687 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:11.261537 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.261508 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.266024 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.265999 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4b5c4ac5-openshift-default-dockercfg-nj8vh\"" Jul 06 19:59:11.287175 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.287145 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:11.311456 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.311420 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:11.374108 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374010 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374108 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374052 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4s5\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374162 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374240 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374264 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374347 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374326 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374389 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374444 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.374523 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.374505 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.391856 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.391827 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:11.475422 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475604 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475432 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475604 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475470 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475604 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475494 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475604 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475534 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475604 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475558 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4s5\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475861 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475714 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475861 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475785 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.475861 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475814 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.476052 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475960 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.476052 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.475990 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.476198 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.476174 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.476267 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.476214 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.476267 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.476216 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.477962 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.477939 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.478234 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.478217 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.484206 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.484182 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.484390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.484370 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4s5\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5\") pod \"e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.573248 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.573202 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:11.712489 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.712463 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:11.714111 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:11.714083 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9cd943_8e37_47a8_bda4_ab932e601563.slice/crio-ba6d5eb4828b2702d8b2c1a11fcfc3fa01c0b307c15ada79107967a038bb625c WatchSource:0}: Error finding container ba6d5eb4828b2702d8b2c1a11fcfc3fa01c0b307c15ada79107967a038bb625c: Status 404 returned error can't find the container with id ba6d5eb4828b2702d8b2c1a11fcfc3fa01c0b307c15ada79107967a038bb625c Jul 06 19:59:11.716168 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.716128 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:11.716242 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.716194 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:11.716242 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:11.716222 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:12.493426 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.493382 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.571292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.571257 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:12.575322 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.575296 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.577201 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.577175 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.577337 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.577296 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-4b5c4ac5-serving-cert\"" Jul 06 19:59:12.586083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.586061 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:12.667483 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.667448 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.673429 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.673404 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" event={"ID":"db9cd943-8e37-47a8-bda4-ab932e601563","Type":"ContainerStarted","Data":"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407"} Jul 06 19:59:12.673570 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.673449 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" event={"ID":"db9cd943-8e37-47a8-bda4-ab932e601563","Type":"ContainerStarted","Data":"ba6d5eb4828b2702d8b2c1a11fcfc3fa01c0b307c15ada79107967a038bb625c"} Jul 06 19:59:12.687034 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.687009 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.687159 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.687124 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.687233 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.687163 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx85b\" (UniqueName: \"kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.697873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.697831 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" podStartSLOduration=1.697817717 podStartE2EDuration="1.697817717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:12.695607984 +0000 UTC m=+1588.043251159" watchObservedRunningTime="2026-07-06 19:59:12.697817717 +0000 UTC m=+1588.045460893" Jul 06 19:59:12.765286 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.765194 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.788385 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.788350 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.788385 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.788391 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx85b\" (UniqueName: \"kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.788648 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.788487 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.789096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.789073 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.790856 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.790838 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.802839 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.802813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx85b\" (UniqueName: \"kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b\") pod \"maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:12.888205 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:12.888167 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:13.045637 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.045442 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:13.048300 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:13.048274 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c608ee_fab2_4544_a0dc_4f570153fe1d.slice/crio-6b99dbaef334fafaa0c3a5e186c74306f979d60e5be2beea59eaa51323ed3ace WatchSource:0}: Error finding container 6b99dbaef334fafaa0c3a5e186c74306f979d60e5be2beea59eaa51323ed3ace: Status 404 returned error can't find the container with id 6b99dbaef334fafaa0c3a5e186c74306f979d60e5be2beea59eaa51323ed3ace Jul 06 19:59:13.049587 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.049568 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:59:13.167870 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.167833 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:13.270228 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.270197 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:13.574250 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.574220 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:13.578969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.578944 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:13.681224 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.681185 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" event={"ID":"25c608ee-fab2-4544-a0dc-4f570153fe1d","Type":"ContainerStarted","Data":"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766"} Jul 06 19:59:13.681421 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.681254 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" event={"ID":"25c608ee-fab2-4544-a0dc-4f570153fe1d","Type":"ContainerStarted","Data":"6b99dbaef334fafaa0c3a5e186c74306f979d60e5be2beea59eaa51323ed3ace"} Jul 06 19:59:13.681533 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.681513 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:13.682461 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.682439 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:13.699840 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:13.699736 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" podStartSLOduration=1.3098682400000001 podStartE2EDuration="1.699721441s" totalImagesPullingTime="389.853201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:13.698609704 +0000 UTC m=+1589.046252873" watchObservedRunningTime="2026-07-06 19:59:13.699721441 +0000 UTC m=+1589.047364643" Jul 06 19:59:14.686900 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:14.686871 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:15.261921 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:15.261855 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:17.568046 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:17.568001 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:17.732503 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:17.732466 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:17.732784 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:17.732749 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" podUID="db9cd943-8e37-47a8-bda4-ab932e601563" containerName="istio-proxy" containerID="cri-o://5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407" gracePeriod=30 Jul 06 19:59:17.771490 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:17.771458 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:17.853122 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:17.853034 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:18.114972 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.114881 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:18.115129 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.115088 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" podUID="25c608ee-fab2-4544-a0dc-4f570153fe1d" containerName="maas-api" containerID="cri-o://60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766" gracePeriod=30 Jul 06 19:59:18.120502 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.120482 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:18.143163 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.143138 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:18.363096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.363070 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:18.439458 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.439430 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx85b\" (UniqueName: \"kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b\") pod \"25c608ee-fab2-4544-a0dc-4f570153fe1d\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " Jul 06 19:59:18.439625 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.439549 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls\") pod \"25c608ee-fab2-4544-a0dc-4f570153fe1d\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " Jul 06 19:59:18.439625 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.439566 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca\") pod \"25c608ee-fab2-4544-a0dc-4f570153fe1d\" (UID: \"25c608ee-fab2-4544-a0dc-4f570153fe1d\") " Jul 06 19:59:18.440025 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.439998 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25c608ee-fab2-4544-a0dc-4f570153fe1d" (UID: "25c608ee-fab2-4544-a0dc-4f570153fe1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:18.441458 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.441433 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b" (OuterVolumeSpecName: "kube-api-access-qx85b") pod "25c608ee-fab2-4544-a0dc-4f570153fe1d" (UID: "25c608ee-fab2-4544-a0dc-4f570153fe1d"). InnerVolumeSpecName "kube-api-access-qx85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:18.441606 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.441589 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25c608ee-fab2-4544-a0dc-4f570153fe1d" (UID: "25c608ee-fab2-4544-a0dc-4f570153fe1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:18.540808 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.540774 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c608ee-fab2-4544-a0dc-4f570153fe1d-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:18.540808 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.540804 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c608ee-fab2-4544-a0dc-4f570153fe1d-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:18.540808 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.540815 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx85b\" (UniqueName: \"kubernetes.io/projected/25c608ee-fab2-4544-a0dc-4f570153fe1d-kube-api-access-qx85b\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:18.709750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.709655 2584 generic.go:356] "Generic (PLEG): container finished" podID="25c608ee-fab2-4544-a0dc-4f570153fe1d" containerID="60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766" exitCode=0 Jul 06 19:59:18.709750 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.709708 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" Jul 06 19:59:18.710292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.709743 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" event={"ID":"25c608ee-fab2-4544-a0dc-4f570153fe1d","Type":"ContainerDied","Data":"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766"} Jul 06 19:59:18.710292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.709802 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w" event={"ID":"25c608ee-fab2-4544-a0dc-4f570153fe1d","Type":"ContainerDied","Data":"6b99dbaef334fafaa0c3a5e186c74306f979d60e5be2beea59eaa51323ed3ace"} Jul 06 19:59:18.710292 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.709827 2584 scope.go:117] "RemoveContainer" containerID="60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766" Jul 06 19:59:18.718802 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.718788 2584 scope.go:117] "RemoveContainer" containerID="60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766" Jul 06 19:59:18.719081 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:18.719061 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766\": container with ID starting with 60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766 not found: ID does not exist" containerID="60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766" Jul 06 19:59:18.719173 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.719089 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766"} err="failed to get container status \"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766\": rpc error: code = NotFound desc = could not find container \"60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766\": container with ID starting with 60286d322d017a2dd233e95fee894b3b1c20db02e8cdca52ecc9435fa3848766 not found: ID does not exist" Jul 06 19:59:18.742382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.742349 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:18.752405 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:18.752379 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-4b5c4ac5-7fd55ddc67-jh27w"] Jul 06 19:59:19.223792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:19.223761 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c608ee-fab2-4544-a0dc-4f570153fe1d" path="/var/lib/kubelet/pods/25c608ee-fab2-4544-a0dc-4f570153fe1d/volumes" Jul 06 19:59:19.539258 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:19.539227 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:22.990090 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:22.990064 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:23.084390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084304 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084351 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084369 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084390 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084388 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084408 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk4s5\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084478 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084514 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084597 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data" (OuterVolumeSpecName: "istio-data") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084606 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084668 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs\") pod \"db9cd943-8e37-47a8-bda4-ab932e601563\" (UID: \"db9cd943-8e37-47a8-bda4-ab932e601563\") " Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084710 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:23.084736 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084722 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:23.085167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.084989 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:23.085167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085026 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:23.085167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085120 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.085167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085142 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.085167 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085157 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.085371 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085171 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9cd943-8e37-47a8-bda4-ab932e601563-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.085371 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.085186 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.086773 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.086743 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:59:23.086880 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.086825 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token" (OuterVolumeSpecName: "istio-token") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:23.087001 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.086983 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:23.087062 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.087011 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5" (OuterVolumeSpecName: "kube-api-access-gk4s5") pod "db9cd943-8e37-47a8-bda4-ab932e601563" (UID: "db9cd943-8e37-47a8-bda4-ab932e601563"). InnerVolumeSpecName "kube-api-access-gk4s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:23.186280 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.186243 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9cd943-8e37-47a8-bda4-ab932e601563-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.186280 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.186273 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4s5\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-kube-api-access-gk4s5\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.186280 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.186286 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9cd943-8e37-47a8-bda4-ab932e601563-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.186514 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.186300 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9cd943-8e37-47a8-bda4-ab932e601563-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:23.735454 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.735421 2584 generic.go:356] "Generic (PLEG): container finished" podID="db9cd943-8e37-47a8-bda4-ab932e601563" containerID="5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407" exitCode=0 Jul 06 19:59:23.735616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.735470 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" Jul 06 19:59:23.735616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.735509 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" event={"ID":"db9cd943-8e37-47a8-bda4-ab932e601563","Type":"ContainerDied","Data":"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407"} Jul 06 19:59:23.735616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.735567 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665" event={"ID":"db9cd943-8e37-47a8-bda4-ab932e601563","Type":"ContainerDied","Data":"ba6d5eb4828b2702d8b2c1a11fcfc3fa01c0b307c15ada79107967a038bb625c"} Jul 06 19:59:23.735616 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.735590 2584 scope.go:117] "RemoveContainer" containerID="5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407" Jul 06 19:59:23.744254 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.744237 2584 scope.go:117] "RemoveContainer" containerID="5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407" Jul 06 19:59:23.744527 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:23.744504 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407\": container with ID starting with 5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407 not found: ID does not exist" containerID="5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407" Jul 06 19:59:23.744595 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.744533 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407"} err="failed to get container status \"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407\": rpc error: code = NotFound desc = could not find container \"5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407\": container with ID starting with 5cf477c5ac8cb88bdf71222f3c217f12f6d65119d015744fb72fbfce0f479407 not found: ID does not exist" Jul 06 19:59:23.759713 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.759682 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:23.765398 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:23.765369 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4b5c4ac5-openshift-default-5795db8d57-rr665"] Jul 06 19:59:24.485728 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.485689 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.565873 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.565843 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.681644 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.681614 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:24.682123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682100 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9cd943-8e37-47a8-bda4-ab932e601563" containerName="istio-proxy" Jul 06 19:59:24.682123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682123 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9cd943-8e37-47a8-bda4-ab932e601563" containerName="istio-proxy" Jul 06 19:59:24.682345 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682140 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c608ee-fab2-4544-a0dc-4f570153fe1d" containerName="maas-api" Jul 06 19:59:24.682345 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682148 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c608ee-fab2-4544-a0dc-4f570153fe1d" containerName="maas-api" Jul 06 19:59:24.682507 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682490 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9cd943-8e37-47a8-bda4-ab932e601563" containerName="istio-proxy" Jul 06 19:59:24.682592 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.682510 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c608ee-fab2-4544-a0dc-4f570153fe1d" containerName="maas-api" Jul 06 19:59:24.687086 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.687068 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.689255 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.689232 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8a2aa7a6-openshift-default-dockercfg-g9mcx\"" Jul 06 19:59:24.706414 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.706379 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:24.743240 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.743142 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.800724 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800690 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.800956 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800742 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.800956 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800803 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.800956 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800848 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.800956 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800878 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.801156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.800991 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.801156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.801019 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.801156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.801100 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772g4\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.801156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.801136 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.818182 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.818154 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.902164 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902131 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902164 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902169 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902200 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772g4\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902401 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902425 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902617 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902501 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902763 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902645 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902763 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902733 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.902994 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.902965 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.903104 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.903035 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.903104 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.903079 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.904660 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.904635 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.904832 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.904816 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.915439 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.915410 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.916048 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.916032 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772g4\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4\") pod \"e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:24.998798 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:24.998715 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:25.136609 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.136577 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:25.140050 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:25.140021 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d9e3ab_98ba_4c92_8a8d_d8ecaa6314b8.slice/crio-b333ac4da6da0ee2438d5f5fb0a0a21198da278d8a8cd06e3438ea19bba27bd2 WatchSource:0}: Error finding container b333ac4da6da0ee2438d5f5fb0a0a21198da278d8a8cd06e3438ea19bba27bd2: Status 404 returned error can't find the container with id b333ac4da6da0ee2438d5f5fb0a0a21198da278d8a8cd06e3438ea19bba27bd2 Jul 06 19:59:25.142133 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.142102 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:25.142212 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.142169 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:25.142212 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.142197 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:25.225682 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.225654 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9cd943-8e37-47a8-bda4-ab932e601563" path="/var/lib/kubelet/pods/db9cd943-8e37-47a8-bda4-ab932e601563/volumes" Jul 06 19:59:25.749408 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.749368 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" event={"ID":"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8","Type":"ContainerStarted","Data":"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab"} Jul 06 19:59:25.749874 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.749435 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" event={"ID":"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8","Type":"ContainerStarted","Data":"b333ac4da6da0ee2438d5f5fb0a0a21198da278d8a8cd06e3438ea19bba27bd2"} Jul 06 19:59:25.955366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.955332 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:25.999705 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:25.999615 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:26.013449 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.013386 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" podStartSLOduration=2.013361728 podStartE2EDuration="2.013361728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:25.796212518 +0000 UTC m=+1601.143855703" watchObservedRunningTime="2026-07-06 19:59:26.013361728 +0000 UTC m=+1601.361004906" Jul 06 19:59:26.015909 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.015878 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:26.019607 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.019584 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.023871 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.023844 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8a2aa7a6-serving-cert\"" Jul 06 19:59:26.031850 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.031824 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:26.057495 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.057446 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.116919 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.116882 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgjj\" (UniqueName: \"kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.117083 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.116942 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.117123 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.117078 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.143874 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.143844 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.217918 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.217884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.218133 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.218028 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgjj\" (UniqueName: \"kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.218133 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.218061 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.218253 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:26.218192 2584 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-8a2aa7a6-serving-cert: secret "maas-api-e2e-ait-8a2aa7a6-serving-cert" not found Jul 06 19:59:26.218316 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:26.218262 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls podName:2ccb7175-afd7-47ac-9d93-9e8b2e64a51d nodeName:}" failed. No retries permitted until 2026-07-06 19:59:26.71823937 +0000 UTC m=+1602.065882541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls") pod "maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" (UID: "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d") : secret "maas-api-e2e-ait-8a2aa7a6-serving-cert" not found Jul 06 19:59:26.218489 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.218466 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.231684 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.231645 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgjj\" (UniqueName: \"kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.400681 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.400597 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.722601 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.722561 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.725330 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.725302 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") pod \"maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:26.771517 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.771467 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.872214 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.872174 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.931388 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:26.931356 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:27.004666 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.004633 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:27.075185 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.075150 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:27.076133 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:27.076104 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccb7175_afd7_47ac_9d93_9e8b2e64a51d.slice/crio-113ab7eaae405ce796c29d3fbca3233a1b0e93aa6524ed170b78aa034452adcd WatchSource:0}: Error finding container 113ab7eaae405ce796c29d3fbca3233a1b0e93aa6524ed170b78aa034452adcd: Status 404 returned error can't find the container with id 113ab7eaae405ce796c29d3fbca3233a1b0e93aa6524ed170b78aa034452adcd Jul 06 19:59:27.760780 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.760690 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" event={"ID":"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d","Type":"ContainerStarted","Data":"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e"} Jul 06 19:59:27.760780 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.760753 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" event={"ID":"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d","Type":"ContainerStarted","Data":"113ab7eaae405ce796c29d3fbca3233a1b0e93aa6524ed170b78aa034452adcd"} Jul 06 19:59:27.761138 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.760995 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:27.761986 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.761964 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:27.805896 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:27.805805 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" podStartSLOduration=2.419760088 podStartE2EDuration="2.805778513s" totalImagesPullingTime="386.018425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:27.780844764 +0000 UTC m=+1603.128487940" watchObservedRunningTime="2026-07-06 19:59:27.805778513 +0000 UTC m=+1603.153421732" Jul 06 19:59:28.134265 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:28.134230 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:28.766004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:28.765975 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:30.931290 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:30.931253 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:31.304621 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.304529 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:31.304783 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.304719 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" podUID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" containerName="maas-api" containerID="cri-o://1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e" gracePeriod=30 Jul 06 19:59:31.328171 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.328113 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:31.559251 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.559183 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:31.673049 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.673013 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwgjj\" (UniqueName: \"kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj\") pod \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " Jul 06 19:59:31.673049 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.673056 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") pod \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " Jul 06 19:59:31.673325 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.673076 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca\") pod \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\" (UID: \"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d\") " Jul 06 19:59:31.673556 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.673522 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" (UID: "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:31.675219 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.675196 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj" (OuterVolumeSpecName: "kube-api-access-mwgjj") pod "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" (UID: "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d"). InnerVolumeSpecName "kube-api-access-mwgjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:31.675295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.675235 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" (UID: "2ccb7175-afd7-47ac-9d93-9e8b2e64a51d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:31.774911 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.774867 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwgjj\" (UniqueName: \"kubernetes.io/projected/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-kube-api-access-mwgjj\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:31.774911 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.774902 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:31.774911 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.774915 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:31.783127 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.783096 2584 generic.go:356] "Generic (PLEG): container finished" podID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" containerID="1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e" exitCode=0 Jul 06 19:59:31.783284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.783143 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" Jul 06 19:59:31.783284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.783184 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" event={"ID":"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d","Type":"ContainerDied","Data":"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e"} Jul 06 19:59:31.783284 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.783267 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw" event={"ID":"2ccb7175-afd7-47ac-9d93-9e8b2e64a51d","Type":"ContainerDied","Data":"113ab7eaae405ce796c29d3fbca3233a1b0e93aa6524ed170b78aa034452adcd"} Jul 06 19:59:31.783397 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.783295 2584 scope.go:117] "RemoveContainer" containerID="1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e" Jul 06 19:59:31.794361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.794338 2584 scope.go:117] "RemoveContainer" containerID="1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e" Jul 06 19:59:31.794851 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:31.794826 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e\": container with ID starting with 1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e not found: ID does not exist" containerID="1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e" Jul 06 19:59:31.794966 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.794865 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e"} err="failed to get container status \"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e\": rpc error: code = NotFound desc = could not find container \"1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e\": container with ID starting with 1b45c344655cac595fec89d591e25bf54e00f1d7312e7cb250f31173c12acf0e not found: ID does not exist" Jul 06 19:59:31.808792 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.808768 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:31.820128 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:31.820059 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8a2aa7a6-59dcb878bc-96wcw"] Jul 06 19:59:33.223765 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:33.223722 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" path="/var/lib/kubelet/pods/2ccb7175-afd7-47ac-9d93-9e8b2e64a51d/volumes" Jul 06 19:59:36.981017 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:36.980919 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:36.981409 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:36.981124 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" podUID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" containerName="istio-proxy" containerID="cri-o://32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab" gracePeriod=30 Jul 06 19:59:37.011421 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:37.011388 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:37.100288 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:37.100257 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:38.777513 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:38.777479 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:42.243146 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.243085 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:42.374911 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.374881 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.374919 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.374986 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375014 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375033 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375203 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375212 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375289 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375317 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375332 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:42.375357 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375345 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-772g4\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4\") pod \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\" (UID: \"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8\") " Jul 06 19:59:42.375659 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375343 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:42.375659 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375447 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:42.375659 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375543 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:42.375816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375718 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.375816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375737 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.375816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375750 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.375816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375761 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.375816 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.375773 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.377366 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.377335 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4" (OuterVolumeSpecName: "kube-api-access-772g4") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "kube-api-access-772g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:42.377480 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.377403 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:59:42.377623 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.377607 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:42.377665 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.377625 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" (UID: "92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:42.476279 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.476250 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.476279 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.476276 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.476279 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.476285 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772g4\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-kube-api-access-772g4\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.476496 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.476294 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:42.836723 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.836690 2584 generic.go:356] "Generic (PLEG): container finished" podID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" containerID="32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab" exitCode=0 Jul 06 19:59:42.836945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.836735 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" event={"ID":"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8","Type":"ContainerDied","Data":"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab"} Jul 06 19:59:42.836945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.836739 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" Jul 06 19:59:42.836945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.836777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd" event={"ID":"92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8","Type":"ContainerDied","Data":"b333ac4da6da0ee2438d5f5fb0a0a21198da278d8a8cd06e3438ea19bba27bd2"} Jul 06 19:59:42.836945 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.836799 2584 scope.go:117] "RemoveContainer" containerID="32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab" Jul 06 19:59:42.845905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.845885 2584 scope.go:117] "RemoveContainer" containerID="32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab" Jul 06 19:59:42.846175 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:42.846156 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab\": container with ID starting with 32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab not found: ID does not exist" containerID="32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab" Jul 06 19:59:42.846245 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.846189 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab"} err="failed to get container status \"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab\": rpc error: code = NotFound desc = could not find container \"32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab\": container with ID starting with 32aeba7cfbc24fc599c66360046fe80d67f103079e3a55656fc2e5ec3211f8ab not found: ID does not exist" Jul 06 19:59:42.859326 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.859303 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:42.866415 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:42.866393 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8a2aa7a6-openshift-default-c7bdd6b7c-nxbcd"] Jul 06 19:59:43.223490 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.223460 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" path="/var/lib/kubelet/pods/92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8/volumes" Jul 06 19:59:43.415156 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.415126 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:43.487150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.487071 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:43.603712 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.603682 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:43.604137 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604123 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" containerName="istio-proxy" Jul 06 19:59:43.604199 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604139 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" containerName="istio-proxy" Jul 06 19:59:43.604199 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604148 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" containerName="maas-api" Jul 06 19:59:43.604199 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604153 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" containerName="maas-api" Jul 06 19:59:43.604300 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604224 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ccb7175-afd7-47ac-9d93-9e8b2e64a51d" containerName="maas-api" Jul 06 19:59:43.604300 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.604234 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d9e3ab-98ba-4c92-8a8d-d8ecaa6314b8" containerName="istio-proxy" Jul 06 19:59:43.608903 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.608626 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.610797 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.610777 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c38f573c-openshift-default-dockercfg-7jg54\"" Jul 06 19:59:43.622455 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.622432 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:43.638941 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.638906 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:43.684557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684527 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684557 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684562 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bj6c\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684585 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684604 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684672 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684736 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684770 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684759 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.684977 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.684819 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.773789 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.773700 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:43.785904 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.785875 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.785914 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.785991 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786037 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786055 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bj6c\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786081 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786076 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786301 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786097 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786301 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786153 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786301 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786193 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786465 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786440 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786580 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786525 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786631 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786604 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786762 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786739 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.786901 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.786880 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.788349 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.788320 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.788561 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.788541 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.796739 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.796710 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.797438 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.797416 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bj6c\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c\") pod \"e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:43.919345 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:43.919305 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:44.070877 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.070845 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:44.072506 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:44.072482 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451afbcd_32a7_4d4b_9c33_909339c69a6a.slice/crio-f22c89331cc0a02200cbf7593156e235ab11466895afc2539572d0cb41596a2a WatchSource:0}: Error finding container f22c89331cc0a02200cbf7593156e235ab11466895afc2539572d0cb41596a2a: Status 404 returned error can't find the container with id f22c89331cc0a02200cbf7593156e235ab11466895afc2539572d0cb41596a2a Jul 06 19:59:44.074585 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.074547 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:44.074664 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.074613 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:44.074664 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.074642 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 19:59:44.768521 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.768487 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:44.823382 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.823349 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:44.827539 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.827517 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.829969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.829946 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-c38f573c-serving-cert\"" Jul 06 19:59:44.839740 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.839704 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:44.856991 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.856954 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" event={"ID":"451afbcd-32a7-4d4b-9c33-909339c69a6a","Type":"ContainerStarted","Data":"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a"} Jul 06 19:59:44.857126 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.857014 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" event={"ID":"451afbcd-32a7-4d4b-9c33-909339c69a6a","Type":"ContainerStarted","Data":"f22c89331cc0a02200cbf7593156e235ab11466895afc2539572d0cb41596a2a"} Jul 06 19:59:44.864402 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.864357 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:44.883233 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.883181 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" podStartSLOduration=1.883163345 podStartE2EDuration="1.883163345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:44.881843616 +0000 UTC m=+1620.229486792" watchObservedRunningTime="2026-07-06 19:59:44.883163345 +0000 UTC m=+1620.230806521" Jul 06 19:59:44.897453 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.897418 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjtd\" (UniqueName: \"kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.897575 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.897463 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.897575 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.897548 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.920278 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.920241 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:44.952519 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.952492 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:44.998640 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.998610 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjtd\" (UniqueName: \"kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.998850 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.998658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.998850 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.998705 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:44.998976 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:44.998888 2584 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-c38f573c-serving-cert: secret "maas-api-e2e-derive-c38f573c-serving-cert" not found Jul 06 19:59:44.999023 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:44.998978 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls podName:7c39a4bc-f3e9-4f01-8f4c-041f67726754 nodeName:}" failed. No retries permitted until 2026-07-06 19:59:45.498954552 +0000 UTC m=+1620.846597725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls") pod "maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" (UID: "7c39a4bc-f3e9-4f01-8f4c-041f67726754") : secret "maas-api-e2e-derive-c38f573c-serving-cert" not found Jul 06 19:59:44.999545 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:44.999522 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:45.013394 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.013369 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjtd\" (UniqueName: \"kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:45.077252 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.077161 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:45.416222 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.416186 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:45.502063 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.502030 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:45.504487 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.504461 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") pod \"maas-api-e2e-derive-c38f573c-657db4867c-kpgbl\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:45.525045 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.525015 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:45.741801 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.741705 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:45.885615 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.885459 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:45.887667 ip-10-0-136-184 kubenswrapper[2584]: W0706 19:59:45.887642 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c39a4bc_f3e9_4f01_8f4c_041f67726754.slice/crio-18f091f526763a6ce6d9f3c6b152e74338580feeb548e4da64629fe658b7b92d WatchSource:0}: Error finding container 18f091f526763a6ce6d9f3c6b152e74338580feeb548e4da64629fe658b7b92d: Status 404 returned error can't find the container with id 18f091f526763a6ce6d9f3c6b152e74338580feeb548e4da64629fe658b7b92d Jul 06 19:59:45.924824 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:45.924800 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:46.674589 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.674556 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:46.869361 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.869326 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" event={"ID":"7c39a4bc-f3e9-4f01-8f4c-041f67726754","Type":"ContainerStarted","Data":"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69"} Jul 06 19:59:46.869537 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.869389 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" event={"ID":"7c39a4bc-f3e9-4f01-8f4c-041f67726754","Type":"ContainerStarted","Data":"18f091f526763a6ce6d9f3c6b152e74338580feeb548e4da64629fe658b7b92d"} Jul 06 19:59:46.869680 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.869657 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:46.870464 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.870450 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:46.889858 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:46.889816 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" podStartSLOduration=2.529948728 podStartE2EDuration="2.889802997s" totalImagesPullingTime="359.854269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:46.888008311 +0000 UTC m=+1622.235651487" watchObservedRunningTime="2026-07-06 19:59:46.889802997 +0000 UTC m=+1622.237446172" Jul 06 19:59:47.874905 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:47.874877 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:49.232969 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.232938 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:49.361342 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.361298 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:49.385739 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.385697 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:49.485192 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.485093 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:49.647456 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.647425 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:49.675948 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.674409 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:49.889200 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.889086 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" podUID="451afbcd-32a7-4d4b-9c33-909339c69a6a" containerName="istio-proxy" containerID="cri-o://e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a" gracePeriod=30 Jul 06 19:59:49.889200 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:49.889135 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" podUID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" containerName="maas-api" containerID="cri-o://4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69" gracePeriod=30 Jul 06 19:59:50.155897 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.155869 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:50.245904 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.245873 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca\") pod \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " Jul 06 19:59:50.246295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.246005 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") pod \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " Jul 06 19:59:50.246295 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.246074 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jjtd\" (UniqueName: \"kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd\") pod \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\" (UID: \"7c39a4bc-f3e9-4f01-8f4c-041f67726754\") " Jul 06 19:59:50.246385 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.246294 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c39a4bc-f3e9-4f01-8f4c-041f67726754" (UID: "7c39a4bc-f3e9-4f01-8f4c-041f67726754"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:50.246422 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.246394 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c39a4bc-f3e9-4f01-8f4c-041f67726754-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:50.248103 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.248073 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c39a4bc-f3e9-4f01-8f4c-041f67726754" (UID: "7c39a4bc-f3e9-4f01-8f4c-041f67726754"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:50.248214 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.248128 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd" (OuterVolumeSpecName: "kube-api-access-7jjtd") pod "7c39a4bc-f3e9-4f01-8f4c-041f67726754" (UID: "7c39a4bc-f3e9-4f01-8f4c-041f67726754"). InnerVolumeSpecName "kube-api-access-7jjtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:50.347498 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.347454 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c39a4bc-f3e9-4f01-8f4c-041f67726754-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:50.347498 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.347490 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjtd\" (UniqueName: \"kubernetes.io/projected/7c39a4bc-f3e9-4f01-8f4c-041f67726754-kube-api-access-7jjtd\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:50.895781 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.895745 2584 generic.go:356] "Generic (PLEG): container finished" podID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" containerID="4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69" exitCode=0 Jul 06 19:59:50.896004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.895785 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" Jul 06 19:59:50.896004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.895825 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" event={"ID":"7c39a4bc-f3e9-4f01-8f4c-041f67726754","Type":"ContainerDied","Data":"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69"} Jul 06 19:59:50.896004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.895876 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl" event={"ID":"7c39a4bc-f3e9-4f01-8f4c-041f67726754","Type":"ContainerDied","Data":"18f091f526763a6ce6d9f3c6b152e74338580feeb548e4da64629fe658b7b92d"} Jul 06 19:59:50.896004 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.895897 2584 scope.go:117] "RemoveContainer" containerID="4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69" Jul 06 19:59:50.905756 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.905740 2584 scope.go:117] "RemoveContainer" containerID="4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69" Jul 06 19:59:50.906036 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:50.906018 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69\": container with ID starting with 4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69 not found: ID does not exist" containerID="4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69" Jul 06 19:59:50.906100 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.906043 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69"} err="failed to get container status \"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69\": rpc error: code = NotFound desc = could not find container \"4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69\": container with ID starting with 4b826bbfe21c9318d795131901d4d92ecffd72014e07487ebee930946f1cad69 not found: ID does not exist" Jul 06 19:59:50.919266 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.919243 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:50.926433 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:50.926413 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-c38f573c-657db4867c-kpgbl"] Jul 06 19:59:51.224207 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:51.224177 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" path="/var/lib/kubelet/pods/7c39a4bc-f3e9-4f01-8f4c-041f67726754/volumes" Jul 06 19:59:51.350098 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:51.350063 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:54.847831 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.847806 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:54.898719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898639 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898682 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898719 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898706 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898996 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898759 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898996 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898821 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898996 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898845 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.898996 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898939 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:54.898996 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.898963 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.899276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899049 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bj6c\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.899276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899076 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy\") pod \"451afbcd-32a7-4d4b-9c33-909339c69a6a\" (UID: \"451afbcd-32a7-4d4b-9c33-909339c69a6a\") " Jul 06 19:59:54.899276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899120 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:54.899276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899213 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:54.899276 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899216 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:54.899525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899380 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:54.899525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899400 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451afbcd-32a7-4d4b-9c33-909339c69a6a-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:54.899525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899415 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:54.899525 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899431 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:54.899726 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.899553 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:54.901288 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.901257 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c" (OuterVolumeSpecName: "kube-api-access-7bj6c") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "kube-api-access-7bj6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:54.901418 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.901326 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:54.901522 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.901503 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:59:54.902126 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.902104 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "451afbcd-32a7-4d4b-9c33-909339c69a6a" (UID: "451afbcd-32a7-4d4b-9c33-909339c69a6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:54.918670 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.918644 2584 generic.go:356] "Generic (PLEG): container finished" podID="451afbcd-32a7-4d4b-9c33-909339c69a6a" containerID="e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a" exitCode=0 Jul 06 19:59:54.918777 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.918692 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" Jul 06 19:59:54.918777 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.918727 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" event={"ID":"451afbcd-32a7-4d4b-9c33-909339c69a6a","Type":"ContainerDied","Data":"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a"} Jul 06 19:59:54.918854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.918783 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f" event={"ID":"451afbcd-32a7-4d4b-9c33-909339c69a6a","Type":"ContainerDied","Data":"f22c89331cc0a02200cbf7593156e235ab11466895afc2539572d0cb41596a2a"} Jul 06 19:59:54.918854 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.918812 2584 scope.go:117] "RemoveContainer" containerID="e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a" Jul 06 19:59:54.927827 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.927810 2584 scope.go:117] "RemoveContainer" containerID="e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a" Jul 06 19:59:54.928097 ip-10-0-136-184 kubenswrapper[2584]: E0706 19:59:54.928076 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a\": container with ID starting with e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a not found: ID does not exist" containerID="e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a" Jul 06 19:59:54.928150 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.928104 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a"} err="failed to get container status \"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a\": rpc error: code = NotFound desc = could not find container \"e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a\": container with ID starting with e0b0a104e6b3da224000a58b5f8b7a0bb39d2c645fe6881102da8252ef17247a not found: ID does not exist" Jul 06 19:59:54.945936 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.945902 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:54.954030 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:54.954008 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c38f573c-openshift-default-7bc978d55f-86w4f"] Jul 06 19:59:55.000141 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.000104 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bj6c\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-kube-api-access-7bj6c\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:55.000141 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.000134 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:55.000141 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.000144 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:55.000141 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.000152 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:55.000399 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.000160 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451afbcd-32a7-4d4b-9c33-909339c69a6a-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 19:59:55.223988 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.223955 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451afbcd-32a7-4d4b-9c33-909339c69a6a" path="/var/lib/kubelet/pods/451afbcd-32a7-4d4b-9c33-909339c69a6a/volumes" Jul 06 19:59:55.480130 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.480015 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:55.577978 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.577944 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:55.990981 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:55.990915 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:56.110136 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:56.110075 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:56.210221 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:56.210162 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:57.593096 ip-10-0-136-184 kubenswrapper[2584]: I0706 19:59:57.593052 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:00.134820 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.134787 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-pvrhh"] Jul 06 20:00:00.135321 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135220 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" containerName="maas-api" Jul 06 20:00:00.135321 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135233 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" containerName="maas-api" Jul 06 20:00:00.135321 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135242 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451afbcd-32a7-4d4b-9c33-909339c69a6a" containerName="istio-proxy" Jul 06 20:00:00.135321 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135248 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="451afbcd-32a7-4d4b-9c33-909339c69a6a" containerName="istio-proxy" Jul 06 20:00:00.135452 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135329 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="451afbcd-32a7-4d4b-9c33-909339c69a6a" containerName="istio-proxy" Jul 06 20:00:00.135452 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.135338 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c39a4bc-f3e9-4f01-8f4c-041f67726754" containerName="maas-api" Jul 06 20:00:00.139764 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.139748 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:00.147133 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.147113 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-pvrhh"] Jul 06 20:00:00.247513 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.247475 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnj8b\" (UniqueName: \"kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b\") pod \"maas-api-key-cleanup-29722800-pvrhh\" (UID: \"a7fc445c-d471-4536-9ca8-0db8d970d1d3\") " pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:00.348543 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.348506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnj8b\" (UniqueName: \"kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b\") pod \"maas-api-key-cleanup-29722800-pvrhh\" (UID: \"a7fc445c-d471-4536-9ca8-0db8d970d1d3\") " pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:00.357348 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.357317 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnj8b\" (UniqueName: \"kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b\") pod \"maas-api-key-cleanup-29722800-pvrhh\" (UID: \"a7fc445c-d471-4536-9ca8-0db8d970d1d3\") " pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:00.451063 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.451028 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:00.793298 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.792990 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-pvrhh"] Jul 06 20:00:00.795814 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:00:00.795775 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fc445c_d471_4536_9ca8_0db8d970d1d3.slice/crio-4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e WatchSource:0}: Error finding container 4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e: Status 404 returned error can't find the container with id 4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e Jul 06 20:00:00.955091 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:00.955058 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" event={"ID":"a7fc445c-d471-4536-9ca8-0db8d970d1d3","Type":"ContainerStarted","Data":"4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e"} Jul 06 20:00:01.961895 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:01.961852 2584 generic.go:356] "Generic (PLEG): container finished" podID="a7fc445c-d471-4536-9ca8-0db8d970d1d3" containerID="addd416a979f696b2af594ba37d2277eec5d98df5fd32081c8bd309a6f3b86e9" exitCode=0 Jul 06 20:00:01.962329 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:01.961914 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" event={"ID":"a7fc445c-d471-4536-9ca8-0db8d970d1d3","Type":"ContainerDied","Data":"addd416a979f696b2af594ba37d2277eec5d98df5fd32081c8bd309a6f3b86e9"} Jul 06 20:00:03.111680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.111652 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:03.174110 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.174072 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnj8b\" (UniqueName: \"kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b\") pod \"a7fc445c-d471-4536-9ca8-0db8d970d1d3\" (UID: \"a7fc445c-d471-4536-9ca8-0db8d970d1d3\") " Jul 06 20:00:03.176236 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.176208 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b" (OuterVolumeSpecName: "kube-api-access-lnj8b") pod "a7fc445c-d471-4536-9ca8-0db8d970d1d3" (UID: "a7fc445c-d471-4536-9ca8-0db8d970d1d3"). InnerVolumeSpecName "kube-api-access-lnj8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:03.275147 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.275107 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnj8b\" (UniqueName: \"kubernetes.io/projected/a7fc445c-d471-4536-9ca8-0db8d970d1d3-kube-api-access-lnj8b\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:00:03.646643 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.646528 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:03.901322 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.901239 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:03.980120 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.980089 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" Jul 06 20:00:03.980293 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.980093 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-pvrhh" event={"ID":"a7fc445c-d471-4536-9ca8-0db8d970d1d3","Type":"ContainerDied","Data":"4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e"} Jul 06 20:00:03.980293 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:03.980231 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4237d1bafce17aa5c306f5f7782e3cb78d18d5f969eaf82e0e48a4bddf42dc6e" Jul 06 20:00:04.023091 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:04.023059 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:04.348427 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:04.348391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:30.317175 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:30.317137 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:36.432877 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:36.432840 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:36.504387 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:36.504349 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:36.600764 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:36.600732 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:47.783046 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:47.783011 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:47.857668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:47.857636 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:52.508138 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:52.508082 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:58.806340 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:58.806307 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:58.907574 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:58.907539 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:59.007636 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:00:59.007604 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:09.527234 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:09.527155 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:10.326524 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:10.326490 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:16.526117 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:16.526065 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:16.628148 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:16.628108 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:16.728236 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:16.728204 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:16.828858 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:16.828768 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:17.596160 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:17.596115 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:17.692213 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:17.692179 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:17.802331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:17.802301 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:19.406359 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:19.406315 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:25.507386 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:25.507350 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:25.583316 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:25.583287 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.120060 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:26.120011 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:32.460685 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:32.460649 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:32.528300 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:32.528264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:32.718477 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:32.718384 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:33.023094 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.023012 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:01:33.023397 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.023385 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fc445c-d471-4536-9ca8-0db8d970d1d3" containerName="cleanup" Jul 06 20:01:33.023440 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.023400 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fc445c-d471-4536-9ca8-0db8d970d1d3" containerName="cleanup" Jul 06 20:01:33.023495 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.023485 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fc445c-d471-4536-9ca8-0db8d970d1d3" containerName="cleanup" Jul 06 20:01:33.028830 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.028778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.033821 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.033796 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7805c6b7-openshift-default-dockercfg-9cbbn\"" Jul 06 20:01:33.048346 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.048317 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:01:33.052131 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.052032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:33.106151 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106113 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106171 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106208 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106239 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106268 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106294 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106323 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106531 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdcs\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.106531 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.106455 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.139192 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.139152 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:33.207548 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207510 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207578 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207636 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207663 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207709 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207749 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.207805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208175 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207823 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208175 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.207863 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdcs\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208175 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.208035 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208342 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.208316 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208580 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.208541 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.208672 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.208561 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.209285 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.209258 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.210461 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.210437 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.211468 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.211367 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.216628 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.216607 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.216958 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.216917 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdcs\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs\") pod \"e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.340051 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.339964 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:33.482344 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.482318 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:01:33.483839 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:01:33.483813 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca45624e_27c7_4178_9fb0_83a91b9c00fb.slice/crio-559e72fcf44fe73b6f2dffb9a810227e2aef574c57085d7772dc1a878cb93480 WatchSource:0}: Error finding container 559e72fcf44fe73b6f2dffb9a810227e2aef574c57085d7772dc1a878cb93480: Status 404 returned error can't find the container with id 559e72fcf44fe73b6f2dffb9a810227e2aef574c57085d7772dc1a878cb93480 Jul 06 20:01:33.486057 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.486021 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:33.486122 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.486106 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:33.486167 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:33.486154 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:34.418278 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:34.418241 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerStarted","Data":"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d"} Jul 06 20:01:34.418465 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:34.418291 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerStarted","Data":"559e72fcf44fe73b6f2dffb9a810227e2aef574c57085d7772dc1a878cb93480"} Jul 06 20:01:34.443909 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:34.443857 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podStartSLOduration=1.443839602 podStartE2EDuration="1.443839602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:34.441373836 +0000 UTC m=+1729.789017021" watchObservedRunningTime="2026-07-06 20:01:34.443839602 +0000 UTC m=+1729.791482778" Jul 06 20:01:35.340599 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:35.340563 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:35.345544 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:35.345519 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:35.423141 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:35.423113 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:35.424048 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:35.424029 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:36.371322 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.371291 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:36.450567 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.450537 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:36.454092 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.454077 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.456270 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.456246 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-7805c6b7-serving-cert\"" Jul 06 20:01:36.456564 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.456547 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:36.467538 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.467511 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:36.543594 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.543563 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.543779 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.543647 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgvn\" (UniqueName: \"kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.543779 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.543726 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.561728 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.561701 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:36.644803 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.644717 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.644803 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.644785 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgvn\" (UniqueName: \"kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.645070 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.644829 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.645070 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:01:36.644943 2584 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-7805c6b7-serving-cert: secret "maas-api-e2e-mt-7805c6b7-serving-cert" not found Jul 06 20:01:36.645070 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:01:36.645002 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls podName:4186cc19-e338-4ec8-be98-b68e58106997 nodeName:}" failed. No retries permitted until 2026-07-06 20:01:37.14498323 +0000 UTC m=+1732.492626401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls") pod "maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" (UID: "4186cc19-e338-4ec8-be98-b68e58106997") : secret "maas-api-e2e-mt-7805c6b7-serving-cert" not found Jul 06 20:01:36.645393 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.645372 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.654316 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.654283 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgvn\" (UniqueName: \"kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:36.755415 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:36.755376 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:37.067564 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.067533 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:37.157123 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.157087 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:37.159840 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.159809 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") pod \"maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:37.166286 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.166257 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:37.367472 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.367382 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:37.511200 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:37.511174 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:37.513260 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:01:37.513234 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4186cc19_e338_4ec8_be98_b68e58106997.slice/crio-cd7a83923a258407b60436d93a813b925649e06ee26f9f0844df4c8455d6eb41 WatchSource:0}: Error finding container cd7a83923a258407b60436d93a813b925649e06ee26f9f0844df4c8455d6eb41: Status 404 returned error can't find the container with id cd7a83923a258407b60436d93a813b925649e06ee26f9f0844df4c8455d6eb41 Jul 06 20:01:38.440872 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:38.440837 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" event={"ID":"4186cc19-e338-4ec8-be98-b68e58106997","Type":"ContainerStarted","Data":"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5"} Jul 06 20:01:38.441062 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:38.440895 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" event={"ID":"4186cc19-e338-4ec8-be98-b68e58106997","Type":"ContainerStarted","Data":"cd7a83923a258407b60436d93a813b925649e06ee26f9f0844df4c8455d6eb41"} Jul 06 20:01:38.441062 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:38.440967 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:38.457317 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:38.457273 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" podStartSLOduration=1.991701359 podStartE2EDuration="2.457257227s" totalImagesPullingTime="465.555868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:38.457029444 +0000 UTC m=+1733.804672620" watchObservedRunningTime="2026-07-06 20:01:38.457257227 +0000 UTC m=+1733.804900402" Jul 06 20:01:40.849754 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:40.849718 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:41.540219 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:41.540185 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:41.639241 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:41.639209 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:41.963345 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:41.963310 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:44.451421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:44.451391 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:48.622876 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:48.622837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:48.722653 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:48.722615 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:49.169218 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.169176 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:49.575384 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.575299 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:49.645970 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.645910 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:49.646420 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.646213 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" podUID="4186cc19-e338-4ec8-be98-b68e58106997" containerName="maas-api" containerID="cri-o://7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5" gracePeriod=30 Jul 06 20:01:49.679049 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.679019 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:49.895575 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.895543 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:49.974151 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.974118 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca\") pod \"4186cc19-e338-4ec8-be98-b68e58106997\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " Jul 06 20:01:49.974326 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.974202 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") pod \"4186cc19-e338-4ec8-be98-b68e58106997\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " Jul 06 20:01:49.974408 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.974379 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xgvn\" (UniqueName: \"kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn\") pod \"4186cc19-e338-4ec8-be98-b68e58106997\" (UID: \"4186cc19-e338-4ec8-be98-b68e58106997\") " Jul 06 20:01:49.974547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.974525 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4186cc19-e338-4ec8-be98-b68e58106997" (UID: "4186cc19-e338-4ec8-be98-b68e58106997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:49.974730 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.974706 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4186cc19-e338-4ec8-be98-b68e58106997-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:01:49.976437 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.976411 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn" (OuterVolumeSpecName: "kube-api-access-4xgvn") pod "4186cc19-e338-4ec8-be98-b68e58106997" (UID: "4186cc19-e338-4ec8-be98-b68e58106997"). InnerVolumeSpecName "kube-api-access-4xgvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:49.976566 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:49.976455 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4186cc19-e338-4ec8-be98-b68e58106997" (UID: "4186cc19-e338-4ec8-be98-b68e58106997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:01:50.075993 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.075947 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4186cc19-e338-4ec8-be98-b68e58106997-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:01:50.075993 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.075979 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgvn\" (UniqueName: \"kubernetes.io/projected/4186cc19-e338-4ec8-be98-b68e58106997-kube-api-access-4xgvn\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:01:50.511554 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.511508 2584 generic.go:356] "Generic (PLEG): container finished" podID="4186cc19-e338-4ec8-be98-b68e58106997" containerID="7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5" exitCode=0 Jul 06 20:01:50.511554 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.511559 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" Jul 06 20:01:50.511836 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.511594 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" event={"ID":"4186cc19-e338-4ec8-be98-b68e58106997","Type":"ContainerDied","Data":"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5"} Jul 06 20:01:50.511836 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.511652 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp" event={"ID":"4186cc19-e338-4ec8-be98-b68e58106997","Type":"ContainerDied","Data":"cd7a83923a258407b60436d93a813b925649e06ee26f9f0844df4c8455d6eb41"} Jul 06 20:01:50.511836 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.511674 2584 scope.go:117] "RemoveContainer" containerID="7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5" Jul 06 20:01:50.521148 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.521129 2584 scope.go:117] "RemoveContainer" containerID="7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5" Jul 06 20:01:50.521431 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:01:50.521410 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5\": container with ID starting with 7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5 not found: ID does not exist" containerID="7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5" Jul 06 20:01:50.521524 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.521438 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5"} err="failed to get container status \"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5\": rpc error: code = NotFound desc = could not find container \"7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5\": container with ID starting with 7bc8be4bddda330765b2ea02d5deb42368a44543eaca1e1fc6ffde7fbbb656d5 not found: ID does not exist" Jul 06 20:01:50.534282 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.534254 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:50.538959 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.538911 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-7805c6b7-774bcf7dc-lckhp"] Jul 06 20:01:50.727505 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.727403 2584 patch_prober.go:28] interesting pod/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:49944->10.133.0.49:15021: read: connection reset by peer" start-of-body= Jul 06 20:01:50.727505 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:50.727485 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:49944->10.133.0.49:15021: read: connection reset by peer" Jul 06 20:01:51.223416 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.223384 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4186cc19-e338-4ec8-be98-b68e58106997" path="/var/lib/kubelet/pods/4186cc19-e338-4ec8-be98-b68e58106997/volumes" Jul 06 20:01:51.519352 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.519266 2584 generic.go:356] "Generic (PLEG): container finished" podID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerID="49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d" exitCode=0 Jul 06 20:01:51.519352 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.519323 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerDied","Data":"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d"} Jul 06 20:01:51.519730 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.519716 2584 scope.go:117] "RemoveContainer" containerID="49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d" Jul 06 20:01:51.520758 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.520729 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:51.520833 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.520819 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:51.520883 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:51.520856 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:01:52.526021 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:52.525986 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerStarted","Data":"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16"} Jul 06 20:01:53.340883 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:53.340845 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:01:53.341141 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:53.341116 2584 patch_prober.go:28] interesting pod/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 20:01:53.341260 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:53.341183 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 20:01:54.340807 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:54.340770 2584 patch_prober.go:28] interesting pod/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 20:01:54.341220 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:54.340837 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 20:01:55.341122 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:55.341086 2584 patch_prober.go:28] interesting pod/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 20:01:55.341507 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:55.341140 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 20:01:56.099357 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.099324 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:56.199322 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.199286 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:56.340716 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.340687 2584 patch_prober.go:28] interesting pod/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 20:01:56.340870 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.340737 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 20:01:56.348311 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.348283 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:01:56.348680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.348501 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" containerID="cri-o://4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16" gracePeriod=30 Jul 06 20:01:56.383985 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.383765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:56.589212 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.589161 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:56.735640 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.735610 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:56.915692 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:56.915658 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:57.003668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:01:57.003576 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:01.912636 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.912618 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:02:01.996043 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.995992 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996100 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996124 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996144 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996169 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgdcs\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996194 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996234 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996577 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996291 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996577 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996313 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:01.996577 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996326 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert\") pod \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\" (UID: \"ca45624e-27c7-4178-9fb0-83a91b9c00fb\") " Jul 06 20:02:01.996577 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996474 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:01.996788 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996720 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:01.996788 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996745 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:01.996788 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996767 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:01.996977 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.996827 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:01.997470 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.997443 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:01.998422 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.998392 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs" (OuterVolumeSpecName: "kube-api-access-zgdcs") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "kube-api-access-zgdcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:01.998525 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.998466 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:01.998525 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.998473 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:02:01.998525 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:01.998510 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca45624e-27c7-4178-9fb0-83a91b9c00fb" (UID: "ca45624e-27c7-4178-9fb0-83a91b9c00fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098033 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098077 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098087 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdcs\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-kube-api-access-zgdcs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098098 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098106 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098114 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca45624e-27c7-4178-9fb0-83a91b9c00fb-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.098438 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.098122 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca45624e-27c7-4178-9fb0-83a91b9c00fb-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:02:02.575806 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.575772 2584 generic.go:356] "Generic (PLEG): container finished" podID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerID="4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16" exitCode=0 Jul 06 20:02:02.576012 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.575809 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerDied","Data":"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16"} Jul 06 20:02:02.576012 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.575845 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" Jul 06 20:02:02.576012 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.575876 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8" event={"ID":"ca45624e-27c7-4178-9fb0-83a91b9c00fb","Type":"ContainerDied","Data":"559e72fcf44fe73b6f2dffb9a810227e2aef574c57085d7772dc1a878cb93480"} Jul 06 20:02:02.576012 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.575908 2584 scope.go:117] "RemoveContainer" containerID="4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16" Jul 06 20:02:02.585458 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.585443 2584 scope.go:117] "RemoveContainer" containerID="49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d" Jul 06 20:02:02.593159 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.593142 2584 scope.go:117] "RemoveContainer" containerID="4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16" Jul 06 20:02:02.593387 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:02:02.593372 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16\": container with ID starting with 4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16 not found: ID does not exist" containerID="4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16" Jul 06 20:02:02.593426 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.593398 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16"} err="failed to get container status \"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16\": rpc error: code = NotFound desc = could not find container \"4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16\": container with ID starting with 4bcd9318ad14fd67e4662562329dfeb86ea5882d00b6ebe5c7fa40147d803d16 not found: ID does not exist" Jul 06 20:02:02.593426 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.593413 2584 scope.go:117] "RemoveContainer" containerID="49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d" Jul 06 20:02:02.593601 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:02:02.593587 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d\": container with ID starting with 49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d not found: ID does not exist" containerID="49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d" Jul 06 20:02:02.593638 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.593606 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d"} err="failed to get container status \"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d\": rpc error: code = NotFound desc = could not find container \"49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d\": container with ID starting with 49703fde53cc23e724d7d4f36096419ad418de885fab4f80e4505430c4845e1d not found: ID does not exist" Jul 06 20:02:02.605466 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.605434 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:02:02.608949 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:02.608915 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7805c6b7-openshift-default-58546b498c-r7tk8"] Jul 06 20:02:03.067380 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:03.067346 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.176248 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:03.176210 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.223417 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:03.223383 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" path="/var/lib/kubelet/pods/ca45624e-27c7-4178-9fb0-83a91b9c00fb/volumes" Jul 06 20:02:03.268591 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:03.268544 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.420341 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:03.420298 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.354551 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:09.354512 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.433042 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:09.432994 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.226433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.226397 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.414248 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414215 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:02:16.414598 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414585 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414599 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414610 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414615 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414630 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4186cc19-e338-4ec8-be98-b68e58106997" containerName="maas-api" Jul 06 20:02:16.414646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414635 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4186cc19-e338-4ec8-be98-b68e58106997" containerName="maas-api" Jul 06 20:02:16.414815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414695 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414704 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca45624e-27c7-4178-9fb0-83a91b9c00fb" containerName="istio-proxy" Jul 06 20:02:16.414815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.414712 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4186cc19-e338-4ec8-be98-b68e58106997" containerName="maas-api" Jul 06 20:02:16.419259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.419238 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.421709 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.421680 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5439eb70-openshift-default-dockercfg-phpbv\"" Jul 06 20:02:16.431365 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.431342 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:02:16.461943 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.461896 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.538026 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.537918 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538026 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.537982 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538247 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538087 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqndn\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538247 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538129 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538247 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538150 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538247 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538228 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538456 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538277 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538456 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538342 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.538456 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.538389 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.564587 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.564550 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.639356 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639318 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639389 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqndn\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639415 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639431 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639459 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639481 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639519 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639555 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639576 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639577 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639986 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639868 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639986 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639905 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.639986 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.639975 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.640153 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.640099 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.640417 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.640399 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.641879 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.641861 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.641987 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.641911 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.649176 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.649146 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqndn\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.649385 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.649358 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token\") pod \"e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.731267 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.731228 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:16.871991 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.871965 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:02:16.874048 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:02:16.873990 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d31c8bf_a795_48d8_8259_b2ee4c34340f.slice/crio-dfba3c39b69bb3a21e826c8b0938353b73f9e10e5788c12bc50828bb8423b4ae WatchSource:0}: Error finding container dfba3c39b69bb3a21e826c8b0938353b73f9e10e5788c12bc50828bb8423b4ae: Status 404 returned error can't find the container with id dfba3c39b69bb3a21e826c8b0938353b73f9e10e5788c12bc50828bb8423b4ae Jul 06 20:02:16.876125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.876095 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:16.876225 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.876161 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:16.876225 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:16.876194 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:17.657998 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:17.657961 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerStarted","Data":"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407"} Jul 06 20:02:17.658450 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:17.658019 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerStarted","Data":"dfba3c39b69bb3a21e826c8b0938353b73f9e10e5788c12bc50828bb8423b4ae"} Jul 06 20:02:17.677866 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:17.677803 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" podStartSLOduration=1.677785528 podStartE2EDuration="1.677785528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:17.67679807 +0000 UTC m=+1773.024441245" watchObservedRunningTime="2026-07-06 20:02:17.677785528 +0000 UTC m=+1773.025428707" Jul 06 20:02:17.732451 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:17.732415 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:17.737100 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:17.737069 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:18.662668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:18.662627 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:18.663688 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:18.663667 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:19.625157 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.625122 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.685870 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.685836 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:02:19.688892 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.688872 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.691908 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.691890 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-5439eb70-serving-cert\"" Jul 06 20:02:19.699292 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.699269 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:02:19.729082 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.729042 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.771808 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.771776 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.772030 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.771954 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.772030 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.772019 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qbq\" (UniqueName: \"kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.827658 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.827601 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.872731 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.872694 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.872906 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.872783 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.872906 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.872815 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qbq\" (UniqueName: \"kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.872906 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:02:19.872850 2584 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-5439eb70-serving-cert: secret "maas-api-e2e-mt-5439eb70-serving-cert" not found Jul 06 20:02:19.873074 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:02:19.872955 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls podName:82579841-8bdd-49cc-9509-7a77733831e2 nodeName:}" failed. No retries permitted until 2026-07-06 20:02:20.372906089 +0000 UTC m=+1775.720549241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls") pod "maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" (UID: "82579841-8bdd-49cc-9509-7a77733831e2") : secret "maas-api-e2e-mt-5439eb70-serving-cert" not found Jul 06 20:02:19.873539 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.873521 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.884522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.884459 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qbq\" (UniqueName: \"kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:19.929052 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:19.929019 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:20.256117 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.256074 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:20.356728 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.356695 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:20.378018 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.377976 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:20.380646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.380621 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") pod \"maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:20.602622 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.602525 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:20.754130 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:20.754097 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:02:20.757096 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:02:20.757068 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82579841_8bdd_49cc_9509_7a77733831e2.slice/crio-ab79249171e8d0885cc0e54599b6cc6f161e2b495187cc549ddc9bc4f5cc72d7 WatchSource:0}: Error finding container ab79249171e8d0885cc0e54599b6cc6f161e2b495187cc549ddc9bc4f5cc72d7: Status 404 returned error can't find the container with id ab79249171e8d0885cc0e54599b6cc6f161e2b495187cc549ddc9bc4f5cc72d7 Jul 06 20:02:21.683329 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:21.683290 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" event={"ID":"82579841-8bdd-49cc-9509-7a77733831e2","Type":"ContainerStarted","Data":"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3"} Jul 06 20:02:21.683528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:21.683339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" event={"ID":"82579841-8bdd-49cc-9509-7a77733831e2","Type":"ContainerStarted","Data":"ab79249171e8d0885cc0e54599b6cc6f161e2b495187cc549ddc9bc4f5cc72d7"} Jul 06 20:02:21.683528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:21.683388 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:21.706947 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:21.706871 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" podStartSLOduration=2.35987936 podStartE2EDuration="2.706848071s" totalImagesPullingTime="346.968711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:21.704149156 +0000 UTC m=+1777.051792328" watchObservedRunningTime="2026-07-06 20:02:21.706848071 +0000 UTC m=+1777.054491247" Jul 06 20:02:23.153323 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:23.153274 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:23.236948 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:23.236888 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:24.127364 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:24.127325 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.694273 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:27.694245 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:02:32.739512 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.739481 2584 generic.go:356] "Generic (PLEG): container finished" podID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerID="2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407" exitCode=0 Jul 06 20:02:32.739980 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.739536 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerDied","Data":"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407"} Jul 06 20:02:32.739980 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.739978 2584 scope.go:117] "RemoveContainer" containerID="2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407" Jul 06 20:02:32.740970 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.740943 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:32.741029 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.741013 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:32.741065 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:32.741047 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:02:33.663319 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:33.663278 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:33.746045 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:33.746006 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerStarted","Data":"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802"} Jul 06 20:02:34.732442 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:34.732413 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:34.733603 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:34.733580 2584 patch_prober.go:28] interesting pod/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 20:02:34.733661 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:34.733634 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 20:02:35.732512 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:35.732476 2584 patch_prober.go:28] interesting pod/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 20:02:35.732889 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:35.732534 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 20:02:35.911697 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:35.911659 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:36.731812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:36.731729 2584 patch_prober.go:28] interesting pod/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 20:02:36.731812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:36.731790 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 20:02:37.735119 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:37.735086 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:37.735498 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:37.735346 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:37.735985 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:37.735969 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:02:41.721168 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:41.721128 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:41.827803 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:02:41.827772 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:01.735533 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:01.735489 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:02.230812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.230773 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:03:02.231113 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.231084 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" podUID="82579841-8bdd-49cc-9509-7a77733831e2" containerName="maas-api" containerID="cri-o://ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3" gracePeriod=30 Jul 06 20:03:02.276965 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.276917 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:02.484558 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.484499 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:03:02.586644 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.586604 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca\") pod \"82579841-8bdd-49cc-9509-7a77733831e2\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " Jul 06 20:03:02.586853 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.586698 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") pod \"82579841-8bdd-49cc-9509-7a77733831e2\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " Jul 06 20:03:02.586960 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.586873 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7qbq\" (UniqueName: \"kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq\") pod \"82579841-8bdd-49cc-9509-7a77733831e2\" (UID: \"82579841-8bdd-49cc-9509-7a77733831e2\") " Jul 06 20:03:02.587029 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.586974 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82579841-8bdd-49cc-9509-7a77733831e2" (UID: "82579841-8bdd-49cc-9509-7a77733831e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:03:02.587158 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.587143 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82579841-8bdd-49cc-9509-7a77733831e2-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:03:02.588795 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.588770 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82579841-8bdd-49cc-9509-7a77733831e2" (UID: "82579841-8bdd-49cc-9509-7a77733831e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:03:02.588921 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.588816 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq" (OuterVolumeSpecName: "kube-api-access-r7qbq") pod "82579841-8bdd-49cc-9509-7a77733831e2" (UID: "82579841-8bdd-49cc-9509-7a77733831e2"). InnerVolumeSpecName "kube-api-access-r7qbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:03:02.688528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.688493 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82579841-8bdd-49cc-9509-7a77733831e2-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:03:02.688528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.688523 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qbq\" (UniqueName: \"kubernetes.io/projected/82579841-8bdd-49cc-9509-7a77733831e2-kube-api-access-r7qbq\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:03:02.898080 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.897989 2584 generic.go:356] "Generic (PLEG): container finished" podID="82579841-8bdd-49cc-9509-7a77733831e2" containerID="ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3" exitCode=0 Jul 06 20:03:02.898080 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.898038 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" Jul 06 20:03:02.898527 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.898081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" event={"ID":"82579841-8bdd-49cc-9509-7a77733831e2","Type":"ContainerDied","Data":"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3"} Jul 06 20:03:02.898527 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.898131 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw" event={"ID":"82579841-8bdd-49cc-9509-7a77733831e2","Type":"ContainerDied","Data":"ab79249171e8d0885cc0e54599b6cc6f161e2b495187cc549ddc9bc4f5cc72d7"} Jul 06 20:03:02.898527 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.898153 2584 scope.go:117] "RemoveContainer" containerID="ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3" Jul 06 20:03:02.907467 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.907449 2584 scope.go:117] "RemoveContainer" containerID="ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3" Jul 06 20:03:02.907745 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:03:02.907725 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3\": container with ID starting with ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3 not found: ID does not exist" containerID="ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3" Jul 06 20:03:02.907795 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.907753 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3"} err="failed to get container status \"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3\": rpc error: code = NotFound desc = could not find container \"ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3\": container with ID starting with ede43a695f0a20dfdaa21b089b2d278ac7b1e3ff3e785ac863bbe2efa50425b3 not found: ID does not exist" Jul 06 20:03:02.920869 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.920839 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:03:02.925659 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:02.925632 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-5439eb70-f7f7dffb6-vtbfw"] Jul 06 20:03:03.222693 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:03.222662 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82579841-8bdd-49cc-9509-7a77733831e2" path="/var/lib/kubelet/pods/82579841-8bdd-49cc-9509-7a77733831e2/volumes" Jul 06 20:03:07.808464 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:07.808435 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:12.874980 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:12.874923 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:49.032724 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:49.032686 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:50.390732 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:50.390697 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:50.802084 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:50.802048 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:56.178006 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.177974 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:56.257998 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.257963 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:56.447680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.447593 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:03:56.447855 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.447832 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" containerID="cri-o://b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802" gracePeriod=30 Jul 06 20:03:56.462329 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.462282 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:56.688207 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:56.688171 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:57.810849 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:57.810815 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:58.236440 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:58.236405 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:58.717973 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:58.717916 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:58.821357 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:58.821328 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:58.917355 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:03:58.917322 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:00.336198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:00.336162 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:01.708096 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.708029 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:04:01.743797 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743762 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.743992 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743830 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.743992 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743874 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.743992 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743908 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.743992 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743959 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.744226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.743998 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.744226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744028 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.744226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744061 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.744226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744092 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:01.744226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744156 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqndn\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn\") pod \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\" (UID: \"2d31c8bf-a795-48d8-8259-b2ee4c34340f\") " Jul 06 20:04:01.744476 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744301 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:01.744476 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744403 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:01.744476 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744456 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:04:01.744623 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744538 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.744623 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744558 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.744623 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744572 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.744623 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744585 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.745009 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.744988 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:01.746356 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.746335 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:04:01.746776 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.746757 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:01.746850 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.746769 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:04:01.746850 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.746844 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn" (OuterVolumeSpecName: "kube-api-access-lqndn") pod "2d31c8bf-a795-48d8-8259-b2ee4c34340f" (UID: "2d31c8bf-a795-48d8-8259-b2ee4c34340f"). InnerVolumeSpecName "kube-api-access-lqndn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:04:01.845403 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.845363 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqndn\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-kube-api-access-lqndn\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.845403 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.845398 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.845403 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.845407 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.845403 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.845414 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.845673 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:01.845422 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d31c8bf-a795-48d8-8259-b2ee4c34340f-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:04:02.196372 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.196339 2584 generic.go:356] "Generic (PLEG): container finished" podID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerID="b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802" exitCode=0 Jul 06 20:04:02.196549 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.196389 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" Jul 06 20:04:02.196549 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.196435 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerDied","Data":"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802"} Jul 06 20:04:02.196549 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.196495 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s" event={"ID":"2d31c8bf-a795-48d8-8259-b2ee4c34340f","Type":"ContainerDied","Data":"dfba3c39b69bb3a21e826c8b0938353b73f9e10e5788c12bc50828bb8423b4ae"} Jul 06 20:04:02.196549 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.196519 2584 scope.go:117] "RemoveContainer" containerID="b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802" Jul 06 20:04:02.206213 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.206196 2584 scope.go:117] "RemoveContainer" containerID="2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407" Jul 06 20:04:02.214451 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.214431 2584 scope.go:117] "RemoveContainer" containerID="b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802" Jul 06 20:04:02.214745 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:04:02.214724 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802\": container with ID starting with b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802 not found: ID does not exist" containerID="b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802" Jul 06 20:04:02.214817 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.214759 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802"} err="failed to get container status \"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802\": rpc error: code = NotFound desc = could not find container \"b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802\": container with ID starting with b5ef2a2d0fafd7c01c7acc23af162d26d4122b625a13529fc1129d9e6fcf3802 not found: ID does not exist" Jul 06 20:04:02.214817 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.214788 2584 scope.go:117] "RemoveContainer" containerID="2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407" Jul 06 20:04:02.215053 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:04:02.215035 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407\": container with ID starting with 2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407 not found: ID does not exist" containerID="2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407" Jul 06 20:04:02.215105 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.215060 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407"} err="failed to get container status \"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407\": rpc error: code = NotFound desc = could not find container \"2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407\": container with ID starting with 2516172bb2df1544728c5eae7ff0246ceb740d348c33734f62cc936de7ba8407 not found: ID does not exist" Jul 06 20:04:02.220250 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.220225 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:04:02.227520 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:02.227491 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5439eb70-openshift-default-6fb95f94ff-bbh7s"] Jul 06 20:04:03.222853 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:03.222818 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" path="/var/lib/kubelet/pods/2d31c8bf-a795-48d8-8259-b2ee4c34340f/volumes" Jul 06 20:04:06.172453 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:06.172412 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:06.251439 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:06.251397 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:06.836658 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:06.836622 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:12.888657 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:12.888622 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:12.993646 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:12.993613 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:13.094601 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:13.094561 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:23.644040 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:23.644007 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:24.167331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:24.167297 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:45.211137 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:45.211105 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:51.806768 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:51.806733 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:51.910892 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:51.910862 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:52.267131 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.267097 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:52.620782 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.620694 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:04:52.621144 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621130 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:52.621201 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621146 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:52.621201 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621161 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82579841-8bdd-49cc-9509-7a77733831e2" containerName="maas-api" Jul 06 20:04:52.621201 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621168 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82579841-8bdd-49cc-9509-7a77733831e2" containerName="maas-api" Jul 06 20:04:52.621318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621241 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:52.621318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621248 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:52.621318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.621254 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82579841-8bdd-49cc-9509-7a77733831e2" containerName="maas-api" Jul 06 20:04:52.624216 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.624201 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.627992 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.627971 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d14fdd-openshift-default-dockercfg-qnmfk\"" Jul 06 20:04:52.642308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.642283 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:04:52.662682 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.662654 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:52.725515 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725521 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725567 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725597 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pjt\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725674 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725707 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725767 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725819 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.725881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.725840 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.826670 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826638 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826684 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826701 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826735 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826889 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.826987 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pjt\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827158 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827352 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827388 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827416 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.827792 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.827592 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.829291 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.829265 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.829496 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.829478 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.836539 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.836517 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.836892 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.836868 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pjt\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt\") pod \"e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.934694 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.934650 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:52.964370 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:52.964340 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:53.082308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.082263 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:04:53.085152 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:04:53.085112 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c812ba_721e_4c49_bce7_6eeb4ada5136.slice/crio-3042e99303312cf7bb16669f889151decab7bcdc5461b3c359943e8b0f200bef WatchSource:0}: Error finding container 3042e99303312cf7bb16669f889151decab7bcdc5461b3c359943e8b0f200bef: Status 404 returned error can't find the container with id 3042e99303312cf7bb16669f889151decab7bcdc5461b3c359943e8b0f200bef Jul 06 20:04:53.087439 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.087408 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:04:53.087528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.087470 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:04:53.087528 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.087497 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:04:53.441240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.441201 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerStarted","Data":"9e0de2631414bfe241c580ba27d6cdd239de129e36581d73411b418603f685a6"} Jul 06 20:04:53.441448 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.441262 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerStarted","Data":"3042e99303312cf7bb16669f889151decab7bcdc5461b3c359943e8b0f200bef"} Jul 06 20:04:53.483712 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.483653 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podStartSLOduration=1.483635943 podStartE2EDuration="1.483635943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:04:53.482513108 +0000 UTC m=+1928.830156275" watchObservedRunningTime="2026-07-06 20:04:53.483635943 +0000 UTC m=+1928.831279111" Jul 06 20:04:53.935244 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:53.935208 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:54.940586 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:54.940555 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:55.451412 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.451379 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:55.452376 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.452360 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:04:55.534493 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.534464 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:55.602280 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.602246 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:04:55.602641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.602628 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:55.602705 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.602643 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d31c8bf-a795-48d8-8259-b2ee4c34340f" containerName="istio-proxy" Jul 06 20:04:55.605869 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.605841 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.608203 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.608175 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-d14fdd-serving-cert\"" Jul 06 20:04:55.624193 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.624169 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:04:55.641658 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.641602 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:55.654671 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.654638 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhz9\" (UniqueName: \"kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.654823 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.654728 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.654879 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.654828 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.726854 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.726743 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:55.756030 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.755997 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhz9\" (UniqueName: \"kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.756229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.756064 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.756229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.756136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.756909 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.756883 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.758812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.758790 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.766494 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.766466 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhz9\" (UniqueName: \"kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9\") pod \"maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:55.916436 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:55.916401 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:56.050969 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:56.050943 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:04:56.052494 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:04:56.052468 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae6ab41_fa56_4f34_b3ec_7199bc5331e1.slice/crio-277047aa266b81d71dc95e600fe94cae97b1ec354e5ce6b3f7b339adc939308b WatchSource:0}: Error finding container 277047aa266b81d71dc95e600fe94cae97b1ec354e5ce6b3f7b339adc939308b: Status 404 returned error can't find the container with id 277047aa266b81d71dc95e600fe94cae97b1ec354e5ce6b3f7b339adc939308b Jul 06 20:04:56.053989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:56.053971 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:04:56.298585 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:56.298500 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:56.391601 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:56.391568 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:56.460180 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:56.460145 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" event={"ID":"dae6ab41-fa56-4f34-b3ec-7199bc5331e1","Type":"ContainerStarted","Data":"277047aa266b81d71dc95e600fe94cae97b1ec354e5ce6b3f7b339adc939308b"} Jul 06 20:04:57.467253 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:57.467215 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" event={"ID":"dae6ab41-fa56-4f34-b3ec-7199bc5331e1","Type":"ContainerStarted","Data":"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287"} Jul 06 20:04:57.467672 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:57.467480 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:04:57.491680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:04:57.491631 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" podStartSLOduration=1.9146461750000001 podStartE2EDuration="2.491616415s" totalImagesPullingTime="576.97024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:04:57.489479516 +0000 UTC m=+1932.837122692" watchObservedRunningTime="2026-07-06 20:04:57.491616415 +0000 UTC m=+1932.839259590" Jul 06 20:05:00.426512 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:00.426474 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:00.774207 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:00.774119 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:01.170536 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.170500 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:05:01.174271 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.174253 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.177228 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.177197 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-91e262-openshift-default-dockercfg-2h46l\"" Jul 06 20:05:01.207019 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.206978 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:05:01.220855 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.220806 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:01.306433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306390 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306437 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306684 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306533 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306684 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306571 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306684 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306598 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pvs\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306684 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306656 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306886 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306886 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306758 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.306886 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.306791 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.407725 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407683 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.407725 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407727 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.407991 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407857 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.407991 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.407991 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407961 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pvs\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408128 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.407989 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408128 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408042 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408128 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408100 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408287 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408287 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408208 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408454 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408608 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408586 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408822 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408802 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.408920 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.408906 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.410071 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.410048 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.410351 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.410331 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.418569 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.418542 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.418694 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.418651 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pvs\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs\") pod \"e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.485124 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.485024 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:01.515360 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.515329 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:01.642768 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.642739 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:05:01.644943 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:05:01.644903 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b4015b_c258_46e7_b5d4_046646b583fa.slice/crio-9458021ca31d449c24c3db56f72eada22b8fd39f75bc54c4126cc7cc3b40b54b WatchSource:0}: Error finding container 9458021ca31d449c24c3db56f72eada22b8fd39f75bc54c4126cc7cc3b40b54b: Status 404 returned error can't find the container with id 9458021ca31d449c24c3db56f72eada22b8fd39f75bc54c4126cc7cc3b40b54b Jul 06 20:05:01.647373 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.647340 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:01.647466 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.647422 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:01.647533 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:01.647467 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:02.499916 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:02.499873 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerStarted","Data":"7cecf169234dd84c4ac4f1467c6d18601a027b73bfbbd953ed0d0d42a6ee90c5"} Jul 06 20:05:02.500339 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:02.499952 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerStarted","Data":"9458021ca31d449c24c3db56f72eada22b8fd39f75bc54c4126cc7cc3b40b54b"} Jul 06 20:05:02.526021 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:02.525964 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podStartSLOduration=1.525944727 podStartE2EDuration="1.525944727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:02.523458374 +0000 UTC m=+1937.871101540" watchObservedRunningTime="2026-07-06 20:05:02.525944727 +0000 UTC m=+1937.873587896" Jul 06 20:05:03.478912 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:03.478884 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:05:03.485746 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:03.485723 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:03.490158 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:03.490139 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:03.508178 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:03.508075 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:03.512671 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:03.512646 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:04.594017 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.593975 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:04.636943 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.636894 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:05:04.640798 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.640780 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.643058 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.643039 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-91e262-serving-cert\"" Jul 06 20:05:04.654874 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.654854 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:05:04.689287 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.689245 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:04.742462 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.742426 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.742664 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.742510 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpvh\" (UniqueName: \"kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.742664 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.742547 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.779596 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.779562 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:04.843119 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.843085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpvh\" (UniqueName: \"kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.843305 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.843139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.843305 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.843179 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.843799 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.843775 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.845642 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.845588 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.862880 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.862847 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpvh\" (UniqueName: \"kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh\") pod \"maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:04.947065 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.947032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:04.954150 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:04.954127 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:05.150562 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:05:05.150521 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca623bf6_de1c_4e5c_a2e5_83289fe57e91.slice/crio-18e31c23e2657bc755142b53e0e47038d705b254501758d738b77df318d40fc9 WatchSource:0}: Error finding container 18e31c23e2657bc755142b53e0e47038d705b254501758d738b77df318d40fc9: Status 404 returned error can't find the container with id 18e31c23e2657bc755142b53e0e47038d705b254501758d738b77df318d40fc9 Jul 06 20:05:05.154514 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:05.154492 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:05:05.468629 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:05.468586 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:05.520423 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:05.520398 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" event={"ID":"ca623bf6-de1c-4e5c-a2e5-83289fe57e91","Type":"ContainerStarted","Data":"18e31c23e2657bc755142b53e0e47038d705b254501758d738b77df318d40fc9"} Jul 06 20:05:05.950259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:05.950228 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:06.527384 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:06.527348 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" event={"ID":"ca623bf6-de1c-4e5c-a2e5-83289fe57e91","Type":"ContainerStarted","Data":"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52"} Jul 06 20:05:06.527600 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:06.527420 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:06.547082 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:06.547031 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" podStartSLOduration=2.195758508 podStartE2EDuration="2.547010663s" totalImagesPullingTime="351.252155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:06.545894526 +0000 UTC m=+1941.893537702" watchObservedRunningTime="2026-07-06 20:05:06.547010663 +0000 UTC m=+1941.894653841" Jul 06 20:05:09.051114 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:09.051077 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:11.418763 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.418733 2584 patch_prober.go:28] interesting pod/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:59178->10.133.0.53:15021: read: connection reset by peer" start-of-body= Jul 06 20:05:11.419167 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.418793 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:59178->10.133.0.53:15021: read: connection reset by peer" Jul 06 20:05:11.552861 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.552828 2584 generic.go:356] "Generic (PLEG): container finished" podID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerID="9e0de2631414bfe241c580ba27d6cdd239de129e36581d73411b418603f685a6" exitCode=0 Jul 06 20:05:11.553039 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.552891 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerDied","Data":"9e0de2631414bfe241c580ba27d6cdd239de129e36581d73411b418603f685a6"} Jul 06 20:05:11.553379 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.553360 2584 scope.go:117] "RemoveContainer" containerID="9e0de2631414bfe241c580ba27d6cdd239de129e36581d73411b418603f685a6" Jul 06 20:05:11.554395 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.554365 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:11.554505 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.554455 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:11.554566 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:11.554524 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:12.537979 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:12.537954 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:05:12.561868 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:12.560986 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerStarted","Data":"c9d76a4bfa3e62fd08e27cc033b0b39561aeb6f3693ba4cf40dfbdbdf57842c8"} Jul 06 20:05:12.935617 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:12.935582 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:05:12.935815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:12.935793 2584 patch_prober.go:28] interesting pod/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:05:12.935862 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:12.935843 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 20:05:13.935074 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:13.935043 2584 patch_prober.go:28] interesting pod/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:05:13.935515 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:13.935489 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 20:05:14.935725 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:14.935684 2584 patch_prober.go:28] interesting pod/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:05:14.936133 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:14.935739 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 20:05:15.935490 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:15.935448 2584 patch_prober.go:28] interesting pod/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 20:05:15.935679 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:15.935526 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 20:05:16.939352 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:16.939314 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:05:16.939769 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:16.939725 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:05:16.940441 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:16.940422 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:05:18.106301 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:18.106268 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:18.240434 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:18.240402 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:18.697322 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:18.696764 2584 patch_prober.go:28] interesting pod/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:56704->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 06 20:05:18.697322 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:18.696851 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:56704->10.133.0.55:15021: read: connection reset by peer" Jul 06 20:05:18.742918 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:18.742887 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:19.615940 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.615902 2584 generic.go:356] "Generic (PLEG): container finished" podID="19b4015b-c258-46e7-b5d4-046646b583fa" containerID="7cecf169234dd84c4ac4f1467c6d18601a027b73bfbbd953ed0d0d42a6ee90c5" exitCode=0 Jul 06 20:05:19.616395 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.615983 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerDied","Data":"7cecf169234dd84c4ac4f1467c6d18601a027b73bfbbd953ed0d0d42a6ee90c5"} Jul 06 20:05:19.616395 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.616386 2584 scope.go:117] "RemoveContainer" containerID="7cecf169234dd84c4ac4f1467c6d18601a027b73bfbbd953ed0d0d42a6ee90c5" Jul 06 20:05:19.617388 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.617362 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:19.617483 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.617428 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:19.617483 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:19.617464 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:05:20.622916 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:20.622875 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerStarted","Data":"934a615154268faa2d3dc82e2432e586a4ec311251eb50675fa82a80d48689d7"} Jul 06 20:05:21.486154 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:21.486101 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:21.486381 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:21.486352 2584 patch_prober.go:28] interesting pod/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:05:21.486440 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:21.486418 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 20:05:22.485390 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:22.485352 2584 patch_prober.go:28] interesting pod/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:05:22.486062 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:22.486014 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 20:05:23.485630 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:23.485580 2584 patch_prober.go:28] interesting pod/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:05:23.486158 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:23.485666 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 20:05:24.485535 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:24.485456 2584 patch_prober.go:28] interesting pod/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 20:05:24.485535 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:24.485529 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 20:05:25.488579 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:25.488550 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:25.489072 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:25.488958 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:25.489645 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:25.489623 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:05:27.057280 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:27.057243 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:27.157147 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:27.157111 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:34.804550 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:34.804513 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:56.036668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:56.036584 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:56.143824 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:05:56.143785 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:06.916949 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:06.916885 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:07.941054 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:07.941023 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:08.262357 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:08.262261 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:08.670148 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:08.670115 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:08.907751 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:08.907723 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:09.008680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.008608 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:09.245097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.245068 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:06:09.245283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.245261 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" podUID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" containerName="maas-api" containerID="cri-o://6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287" gracePeriod=30 Jul 06 20:06:09.285183 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.285153 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:09.502905 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.502881 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:06:09.602259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.602159 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whhz9\" (UniqueName: \"kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9\") pod \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " Jul 06 20:06:09.602259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.602214 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca\") pod \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " Jul 06 20:06:09.602497 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.602308 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls\") pod \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\" (UID: \"dae6ab41-fa56-4f34-b3ec-7199bc5331e1\") " Jul 06 20:06:09.602613 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.602589 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dae6ab41-fa56-4f34-b3ec-7199bc5331e1" (UID: "dae6ab41-fa56-4f34-b3ec-7199bc5331e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:06:09.604308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.604286 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dae6ab41-fa56-4f34-b3ec-7199bc5331e1" (UID: "dae6ab41-fa56-4f34-b3ec-7199bc5331e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:09.604379 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.604323 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9" (OuterVolumeSpecName: "kube-api-access-whhz9") pod "dae6ab41-fa56-4f34-b3ec-7199bc5331e1" (UID: "dae6ab41-fa56-4f34-b3ec-7199bc5331e1"). InnerVolumeSpecName "kube-api-access-whhz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:09.703561 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.703530 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:09.703561 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.703555 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhz9\" (UniqueName: \"kubernetes.io/projected/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-kube-api-access-whhz9\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:09.703561 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.703565 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dae6ab41-fa56-4f34-b3ec-7199bc5331e1-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:09.875843 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.875754 2584 generic.go:356] "Generic (PLEG): container finished" podID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" containerID="6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287" exitCode=0 Jul 06 20:06:09.875843 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.875801 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" Jul 06 20:06:09.875843 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.875827 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" event={"ID":"dae6ab41-fa56-4f34-b3ec-7199bc5331e1","Type":"ContainerDied","Data":"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287"} Jul 06 20:06:09.876130 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.875869 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v" event={"ID":"dae6ab41-fa56-4f34-b3ec-7199bc5331e1","Type":"ContainerDied","Data":"277047aa266b81d71dc95e600fe94cae97b1ec354e5ce6b3f7b339adc939308b"} Jul 06 20:06:09.876130 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.875890 2584 scope.go:117] "RemoveContainer" containerID="6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287" Jul 06 20:06:09.884939 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.884898 2584 scope.go:117] "RemoveContainer" containerID="6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287" Jul 06 20:06:09.885227 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:06:09.885209 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287\": container with ID starting with 6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287 not found: ID does not exist" containerID="6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287" Jul 06 20:06:09.885298 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.885236 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287"} err="failed to get container status \"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287\": rpc error: code = NotFound desc = could not find container \"6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287\": container with ID starting with 6632ab707ea57176803f43a00d0e0859aab4818b2dd60a49b14267602dbca287 not found: ID does not exist" Jul 06 20:06:09.903036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.903008 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:06:09.910575 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:09.910551 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d14fdd-6776c7bcfb-pnt8v"] Jul 06 20:06:11.223394 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:11.223364 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" path="/var/lib/kubelet/pods/dae6ab41-fa56-4f34-b3ec-7199bc5331e1/volumes" Jul 06 20:06:13.727433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:13.727401 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:51.731723 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:51.731676 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:51.975769 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:51.975733 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:06:51.976001 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:51.975976 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" containerID="cri-o://c9d76a4bfa3e62fd08e27cc033b0b39561aeb6f3693ba4cf40dfbdbdf57842c8" gracePeriod=30 Jul 06 20:06:52.008803 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:52.008685 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:52.260018 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:52.259909 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:52.617905 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:52.617791 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:52.878800 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:52.878678 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:53.172853 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.172817 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:53.286579 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.286547 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:53.509066 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.508984 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:06:53.509223 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.509186 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" podUID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" containerName="maas-api" containerID="cri-o://1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52" gracePeriod=30 Jul 06 20:06:53.542860 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.542823 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:53.766102 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.766031 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:06:53.824479 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.824444 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llpvh\" (UniqueName: \"kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh\") pod \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " Jul 06 20:06:53.824651 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.824497 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca\") pod \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " Jul 06 20:06:53.824651 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.824522 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls\") pod \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\" (UID: \"ca623bf6-de1c-4e5c-a2e5-83289fe57e91\") " Jul 06 20:06:53.824838 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.824817 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca623bf6-de1c-4e5c-a2e5-83289fe57e91" (UID: "ca623bf6-de1c-4e5c-a2e5-83289fe57e91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:06:53.826593 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.826570 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca623bf6-de1c-4e5c-a2e5-83289fe57e91" (UID: "ca623bf6-de1c-4e5c-a2e5-83289fe57e91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:53.826697 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.826654 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh" (OuterVolumeSpecName: "kube-api-access-llpvh") pod "ca623bf6-de1c-4e5c-a2e5-83289fe57e91" (UID: "ca623bf6-de1c-4e5c-a2e5-83289fe57e91"). InnerVolumeSpecName "kube-api-access-llpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:53.926034 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.925998 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llpvh\" (UniqueName: \"kubernetes.io/projected/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-kube-api-access-llpvh\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:53.926034 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.926028 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:53.926034 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:53.926039 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca623bf6-de1c-4e5c-a2e5-83289fe57e91-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:54.106702 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.106617 2584 generic.go:356] "Generic (PLEG): container finished" podID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" containerID="1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52" exitCode=0 Jul 06 20:06:54.106702 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.106666 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" Jul 06 20:06:54.106890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.106669 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" event={"ID":"ca623bf6-de1c-4e5c-a2e5-83289fe57e91","Type":"ContainerDied","Data":"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52"} Jul 06 20:06:54.106890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.106795 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk" event={"ID":"ca623bf6-de1c-4e5c-a2e5-83289fe57e91","Type":"ContainerDied","Data":"18e31c23e2657bc755142b53e0e47038d705b254501758d738b77df318d40fc9"} Jul 06 20:06:54.106890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.106826 2584 scope.go:117] "RemoveContainer" containerID="1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52" Jul 06 20:06:54.116012 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.115985 2584 scope.go:117] "RemoveContainer" containerID="1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52" Jul 06 20:06:54.116281 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:06:54.116263 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52\": container with ID starting with 1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52 not found: ID does not exist" containerID="1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52" Jul 06 20:06:54.116347 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.116291 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52"} err="failed to get container status \"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52\": rpc error: code = NotFound desc = could not find container \"1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52\": container with ID starting with 1b46ed54ff995c6a0e1ed8b995d512f917efaa6dce4e6a25c3dbf89d70b8fe52 not found: ID does not exist" Jul 06 20:06:54.132547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.132523 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:06:54.135773 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:54.135750 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-91e262-684ff4fd65-rv9xk"] Jul 06 20:06:55.224673 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:55.224638 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" path="/var/lib/kubelet/pods/ca623bf6-de1c-4e5c-a2e5-83289fe57e91/volumes" Jul 06 20:06:57.130962 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.130911 2584 generic.go:356] "Generic (PLEG): container finished" podID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerID="c9d76a4bfa3e62fd08e27cc033b0b39561aeb6f3693ba4cf40dfbdbdf57842c8" exitCode=0 Jul 06 20:06:57.131356 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.130966 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerDied","Data":"c9d76a4bfa3e62fd08e27cc033b0b39561aeb6f3693ba4cf40dfbdbdf57842c8"} Jul 06 20:06:57.131356 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.131022 2584 scope.go:117] "RemoveContainer" containerID="9e0de2631414bfe241c580ba27d6cdd239de129e36581d73411b418603f685a6" Jul 06 20:06:57.232292 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.232264 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:06:57.361500 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361412 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361500 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361465 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361500 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361495 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361554 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361622 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361688 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7pjt\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361708 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361730 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.361770 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361746 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:57.362161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361757 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token\") pod \"74c812ba-721e-4c49-bce7-6eeb4ada5136\" (UID: \"74c812ba-721e-4c49-bce7-6eeb4ada5136\") " Jul 06 20:06:57.362161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.361861 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:57.362283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.362242 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.362283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.362263 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.362283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.362239 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:57.362283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.362247 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data" (OuterVolumeSpecName: "istio-data") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:57.362572 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.362553 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:06:57.364059 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.364034 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt" (OuterVolumeSpecName: "kube-api-access-m7pjt") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "kube-api-access-m7pjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:57.364452 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.364433 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:06:57.364631 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.364613 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token" (OuterVolumeSpecName: "istio-token") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:57.364884 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.364852 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74c812ba-721e-4c49-bce7-6eeb4ada5136" (UID: "74c812ba-721e-4c49-bce7-6eeb4ada5136"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:57.463458 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463422 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463458 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463453 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463458 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463466 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pjt\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-kube-api-access-m7pjt\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463706 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463478 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463706 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463490 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c812ba-721e-4c49-bce7-6eeb4ada5136-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463706 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463503 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:57.463706 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:57.463513 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c812ba-721e-4c49-bce7-6eeb4ada5136-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:06:58.136698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:58.136666 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" Jul 06 20:06:58.136698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:58.136693 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn" event={"ID":"74c812ba-721e-4c49-bce7-6eeb4ada5136","Type":"ContainerDied","Data":"3042e99303312cf7bb16669f889151decab7bcdc5461b3c359943e8b0f200bef"} Jul 06 20:06:58.137239 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:58.136756 2584 scope.go:117] "RemoveContainer" containerID="c9d76a4bfa3e62fd08e27cc033b0b39561aeb6f3693ba4cf40dfbdbdf57842c8" Jul 06 20:06:58.164025 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:58.163999 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:06:58.169886 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:58.169859 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d14fdd-openshift-default-594df6bfff-44fdn"] Jul 06 20:06:59.223527 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:59.223497 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" path="/var/lib/kubelet/pods/74c812ba-721e-4c49-bce7-6eeb4ada5136/volumes" Jul 06 20:06:59.773650 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:59.773616 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:59.853375 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:06:59.853344 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:00.057129 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:00.057038 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:07:00.057301 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:00.057255 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" containerID="cri-o://934a615154268faa2d3dc82e2432e586a4ec311251eb50675fa82a80d48689d7" gracePeriod=30 Jul 06 20:07:00.062455 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:00.062425 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:00.323127 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:00.323035 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:00.750871 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:00.750838 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:01.049707 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.049631 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:01.050037 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050025 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" containerName="maas-api" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050038 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" containerName="maas-api" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050058 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" containerName="maas-api" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050065 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" containerName="maas-api" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050075 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050080 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.050089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050090 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.050308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050096 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.050308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050147 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae6ab41-fa56-4f34-b3ec-7199bc5331e1" containerName="maas-api" Jul 06 20:07:01.050308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050155 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.050308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050163 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca623bf6-de1c-4e5c-a2e5-83289fe57e91" containerName="maas-api" Jul 06 20:07:01.050308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.050169 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c812ba-721e-4c49-bce7-6eeb4ada5136" containerName="istio-proxy" Jul 06 20:07:01.053348 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.053329 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.055888 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.055867 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4e125-openshift-default-dockercfg-hvkk2\"" Jul 06 20:07:01.067687 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.067663 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:01.090125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.090099 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:01.199253 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199216 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199253 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199254 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199297 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199326 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmdx\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199367 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199388 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199422 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199482 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199465 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.199684 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.199485 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.300972 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301021 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301087 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmdx\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301135 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301176 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301243 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301404 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301268 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301517 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301579 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301646 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301780 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301688 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.301962 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.301911 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.303396 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.303362 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.303624 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.303604 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.310262 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.310242 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.310414 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.310394 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmdx\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx\") pod \"e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.333085 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.333059 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:01.364174 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.364151 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:01.505308 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.505280 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:01.506862 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:07:01.506837 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e97f40_d433_433f_b25e_30d530babc07.slice/crio-9e7a0d6b4df079e97a189d3065ff933cdbf4e083700483338e74c4f2d672f97a WatchSource:0}: Error finding container 9e7a0d6b4df079e97a189d3065ff933cdbf4e083700483338e74c4f2d672f97a: Status 404 returned error can't find the container with id 9e7a0d6b4df079e97a189d3065ff933cdbf4e083700483338e74c4f2d672f97a Jul 06 20:07:01.509046 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.509020 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:01.509116 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.509080 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:01.509116 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:01.509109 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:02.170231 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:02.170186 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerStarted","Data":"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11"} Jul 06 20:07:02.170459 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:02.170243 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerStarted","Data":"9e7a0d6b4df079e97a189d3065ff933cdbf4e083700483338e74c4f2d672f97a"} Jul 06 20:07:02.195237 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:02.195189 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podStartSLOduration=1.195170764 podStartE2EDuration="1.195170764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:02.193255421 +0000 UTC m=+2057.540898596" watchObservedRunningTime="2026-07-06 20:07:02.195170764 +0000 UTC m=+2057.542813940" Jul 06 20:07:02.365138 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:02.365100 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:02.370539 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:02.370475 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:03.176775 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:03.176748 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:03.177696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:03.177675 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:04.266154 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.266101 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:04.393510 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.393479 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:04.396363 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.396345 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.402055 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.402032 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-f4e125-serving-cert\"" Jul 06 20:07:04.426993 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.426960 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:04.440287 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.440251 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:04.534014 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.533886 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xlg\" (UniqueName: \"kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.534014 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.533966 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.534014 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.534012 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.535024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.534997 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:04.634868 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.634825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xlg\" (UniqueName: \"kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.635056 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.634904 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.635056 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.635032 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.635663 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.635639 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.637486 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.637464 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.650093 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.650067 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xlg\" (UniqueName: \"kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg\") pod \"maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.684480 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.684450 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:04.707093 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.707066 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:04.851107 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:07:04.851071 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2806cd_5492_44b6_bf23_0b7836dc29bc.slice/crio-fe7481d7b0461cd23113b97c902b840d363b210f6538dc662a063d1118a0334e WatchSource:0}: Error finding container fe7481d7b0461cd23113b97c902b840d363b210f6538dc662a063d1118a0334e: Status 404 returned error can't find the container with id fe7481d7b0461cd23113b97c902b840d363b210f6538dc662a063d1118a0334e Jul 06 20:07:04.854750 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.854726 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:04.977563 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:04.977530 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:05.070402 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.070366 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:05.175941 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.175904 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:05.191193 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.191163 2584 generic.go:356] "Generic (PLEG): container finished" podID="19b4015b-c258-46e7-b5d4-046646b583fa" containerID="934a615154268faa2d3dc82e2432e586a4ec311251eb50675fa82a80d48689d7" exitCode=0 Jul 06 20:07:05.191339 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.191241 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerDied","Data":"934a615154268faa2d3dc82e2432e586a4ec311251eb50675fa82a80d48689d7"} Jul 06 20:07:05.191339 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.191301 2584 scope.go:117] "RemoveContainer" containerID="7cecf169234dd84c4ac4f1467c6d18601a027b73bfbbd953ed0d0d42a6ee90c5" Jul 06 20:07:05.192632 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.192604 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" event={"ID":"2c2806cd-5492-44b6-bf23-0b7836dc29bc","Type":"ContainerStarted","Data":"fe7481d7b0461cd23113b97c902b840d363b210f6538dc662a063d1118a0334e"} Jul 06 20:07:05.281109 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.281084 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:05.305472 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.305447 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:07:05.444066 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444035 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444117 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444139 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444170 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444194 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444240 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444219 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2pvs\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444247 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444357 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:05.444509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444402 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444462 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:05.444509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444472 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert\") pod \"19b4015b-c258-46e7-b5d4-046646b583fa\" (UID: \"19b4015b-c258-46e7-b5d4-046646b583fa\") " Jul 06 20:07:05.444864 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444840 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:05.445026 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444879 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.445026 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.444900 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.445119 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.445046 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:05.445119 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.445065 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:05.446487 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.446464 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:05.446605 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.446584 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs" (OuterVolumeSpecName: "kube-api-access-w2pvs") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "kube-api-access-w2pvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:05.446764 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.446743 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:05.446764 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.446746 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19b4015b-c258-46e7-b5d4-046646b583fa" (UID: "19b4015b-c258-46e7-b5d4-046646b583fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:07:05.546137 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546101 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546137 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546131 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546137 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546144 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4015b-c258-46e7-b5d4-046646b583fa-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546370 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546158 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pvs\" (UniqueName: \"kubernetes.io/projected/19b4015b-c258-46e7-b5d4-046646b583fa-kube-api-access-w2pvs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546370 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546171 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546370 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546179 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4015b-c258-46e7-b5d4-046646b583fa-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:05.546370 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:05.546187 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4015b-c258-46e7-b5d4-046646b583fa-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:06.199546 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.199516 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" Jul 06 20:07:06.199721 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.199514 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr" event={"ID":"19b4015b-c258-46e7-b5d4-046646b583fa","Type":"ContainerDied","Data":"9458021ca31d449c24c3db56f72eada22b8fd39f75bc54c4126cc7cc3b40b54b"} Jul 06 20:07:06.199721 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.199677 2584 scope.go:117] "RemoveContainer" containerID="934a615154268faa2d3dc82e2432e586a4ec311251eb50675fa82a80d48689d7" Jul 06 20:07:06.201226 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.201200 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" event={"ID":"2c2806cd-5492-44b6-bf23-0b7836dc29bc","Type":"ContainerStarted","Data":"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f"} Jul 06 20:07:06.201362 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.201348 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:06.226708 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.226659 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" podStartSLOduration=1.8466476859999998 podStartE2EDuration="2.226640102s" totalImagesPullingTime="379.992416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:06.22503703 +0000 UTC m=+2061.572680206" watchObservedRunningTime="2026-07-06 20:07:06.226640102 +0000 UTC m=+2061.574283277" Jul 06 20:07:06.243725 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.243695 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:07:06.246137 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:06.246111 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-91e262-openshift-default-77755d64f7-p9vzr"] Jul 06 20:07:07.224241 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:07.224209 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" path="/var/lib/kubelet/pods/19b4015b-c258-46e7-b5d4-046646b583fa/volumes" Jul 06 20:07:08.708489 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:08.708452 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.221341 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.221306 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.549222 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549138 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:09.549588 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549575 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.549641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549590 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.549641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549612 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.549641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549619 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.549749 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549697 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.549749 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.549707 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b4015b-c258-46e7-b5d4-046646b583fa" containerName="istio-proxy" Jul 06 20:07:09.554369 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.554354 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.556500 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.556484 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-618838-openshift-default-dockercfg-76c2x\"" Jul 06 20:07:09.579544 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.579514 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.579813 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.579776 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:09.689563 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689517 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.689785 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689651 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.689863 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689789 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.689956 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.689956 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689901 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbkn\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.690076 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689955 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.690076 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.689998 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.690076 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.690042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.690235 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.690111 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791218 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791179 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791227 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbkn\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791373 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791422 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791458 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791534 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791561 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.791696 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791655 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.792282 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.791906 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.792282 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.792010 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.792282 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.792178 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.792282 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.792260 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.793839 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.793816 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.794041 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.794023 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.801798 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.801730 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.802300 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.802278 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbkn\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn\") pod \"e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:09.876525 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:09.876230 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:10.050901 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.044991 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:10.052989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.052957 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:10.053109 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.053057 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:10.053178 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.053109 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:10.227700 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.227669 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerStarted","Data":"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e"} Jul 06 20:07:10.227866 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.227726 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerStarted","Data":"9c54b36ecc2df150ed5a00aac334efa72557e083df9923117f5267a6f38afc73"} Jul 06 20:07:10.252368 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.252317 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podStartSLOduration=1.252300779 podStartE2EDuration="1.252300779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:10.250114555 +0000 UTC m=+2065.597757727" watchObservedRunningTime="2026-07-06 20:07:10.252300779 +0000 UTC m=+2065.599943954" Jul 06 20:07:10.374263 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.374135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:10.877312 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:10.877219 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:11.886637 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:11.886594 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:12.213494 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.213465 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:12.238014 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.237957 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:12.238970 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.238948 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:12.341259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.341227 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:12.432115 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.432085 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:12.658792 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.658710 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:12.912539 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:12.912503 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:13.412004 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:13.411979 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:13.528837 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:13.528776 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:16.263849 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.263814 2584 generic.go:356] "Generic (PLEG): container finished" podID="b9e97f40-d433-433f-b25e-30d530babc07" containerID="922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11" exitCode=0 Jul 06 20:07:16.264289 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.263883 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerDied","Data":"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11"} Jul 06 20:07:16.264340 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.264299 2584 scope.go:117] "RemoveContainer" containerID="922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11" Jul 06 20:07:16.265340 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.265312 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:16.265401 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.265379 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:16.265438 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.265411 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:16.800318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:16.800282 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:17.271433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:17.271394 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerStarted","Data":"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3"} Jul 06 20:07:17.364611 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:17.364569 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:17.365808 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:17.365780 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 20:07:17.365953 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:17.365845 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 20:07:18.177744 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:18.177699 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:18.365713 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:18.365239 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 20:07:18.365713 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:18.365303 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 20:07:19.365584 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:19.365549 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 20:07:19.365864 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:19.365834 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 20:07:20.364787 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:20.364744 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 20:07:20.365024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:20.364817 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 20:07:21.365665 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:21.365143 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 20:07:21.365665 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:21.365218 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 20:07:22.368811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:22.368776 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:22.369811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:22.369794 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:24.145535 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:24.145505 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:24.243105 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:24.243071 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.010108 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.010079 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.438954 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.438891 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.654706 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.654673 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.755025 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.754918 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.972805 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.972755 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:25.978689 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:25.978649 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" podUID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" containerName="maas-api" containerID="cri-o://01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f" gracePeriod=30 Jul 06 20:07:26.019158 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.019083 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.256966 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.256918 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:26.324335 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.324237 2584 generic.go:356] "Generic (PLEG): container finished" podID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" containerID="01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f" exitCode=0 Jul 06 20:07:26.324335 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.324283 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" Jul 06 20:07:26.324530 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.324281 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" event={"ID":"2c2806cd-5492-44b6-bf23-0b7836dc29bc","Type":"ContainerDied","Data":"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f"} Jul 06 20:07:26.324530 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.324417 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6" event={"ID":"2c2806cd-5492-44b6-bf23-0b7836dc29bc","Type":"ContainerDied","Data":"fe7481d7b0461cd23113b97c902b840d363b210f6538dc662a063d1118a0334e"} Jul 06 20:07:26.324530 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.324447 2584 scope.go:117] "RemoveContainer" containerID="01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f" Jul 06 20:07:26.332903 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.332886 2584 scope.go:117] "RemoveContainer" containerID="01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f" Jul 06 20:07:26.333182 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:07:26.333164 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f\": container with ID starting with 01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f not found: ID does not exist" containerID="01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f" Jul 06 20:07:26.333241 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.333193 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f"} err="failed to get container status \"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f\": rpc error: code = NotFound desc = could not find container \"01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f\": container with ID starting with 01693d7e7a7c79ff09ba1c0bc9e7d2d6eb4f75a327eed566da76ed61d07f686f not found: ID does not exist" Jul 06 20:07:26.367961 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.363268 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca\") pod \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " Jul 06 20:07:26.367961 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.363397 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5xlg\" (UniqueName: \"kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg\") pod \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " Jul 06 20:07:26.367961 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.363450 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls\") pod \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\" (UID: \"2c2806cd-5492-44b6-bf23-0b7836dc29bc\") " Jul 06 20:07:26.372962 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.372454 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c2806cd-5492-44b6-bf23-0b7836dc29bc" (UID: "2c2806cd-5492-44b6-bf23-0b7836dc29bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:26.386951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.383990 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg" (OuterVolumeSpecName: "kube-api-access-n5xlg") pod "2c2806cd-5492-44b6-bf23-0b7836dc29bc" (UID: "2c2806cd-5492-44b6-bf23-0b7836dc29bc"). InnerVolumeSpecName "kube-api-access-n5xlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:26.386951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.386056 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c2806cd-5492-44b6-bf23-0b7836dc29bc" (UID: "2c2806cd-5492-44b6-bf23-0b7836dc29bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:07:26.465097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.464998 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2806cd-5492-44b6-bf23-0b7836dc29bc-openshift-service-ca\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:26.465097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.465042 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xlg\" (UniqueName: \"kubernetes.io/projected/2c2806cd-5492-44b6-bf23-0b7836dc29bc-kube-api-access-n5xlg\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:26.465097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.465070 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2806cd-5492-44b6-bf23-0b7836dc29bc-maas-api-tls\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:26.665552 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.665510 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:26.667178 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:26.667144 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f4e125-79d7b4cd55-dlxc6"] Jul 06 20:07:27.224134 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:27.224103 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" path="/var/lib/kubelet/pods/2c2806cd-5492-44b6-bf23-0b7836dc29bc/volumes" Jul 06 20:07:32.253815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.253708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:32.336954 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.336852 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:32.357155 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.357120 2584 generic.go:356] "Generic (PLEG): container finished" podID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerID="db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e" exitCode=0 Jul 06 20:07:32.357353 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.357192 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerDied","Data":"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e"} Jul 06 20:07:32.357673 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.357656 2584 scope.go:117] "RemoveContainer" containerID="db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e" Jul 06 20:07:32.358744 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.358717 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:32.358856 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.358782 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:32.358856 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.358814 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:32.535777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.535741 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:32.536146 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.535983 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" containerID="cri-o://c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3" gracePeriod=30 Jul 06 20:07:32.552834 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.552798 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:32.764176 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.764135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:32.969326 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:32.969292 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:33.367204 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.367166 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerStarted","Data":"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec"} Jul 06 20:07:33.375436 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.375405 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:33.505654 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.505623 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:33.608001 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.607952 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:33.872708 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.872665 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:33.880349 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.880312 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:33.880645 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.880552 2584 patch_prober.go:28] interesting pod/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 06 20:07:33.880853 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:33.880811 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 06 20:07:34.877793 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:34.877313 2584 patch_prober.go:28] interesting pod/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 06 20:07:34.877793 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:34.877375 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 06 20:07:35.877871 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:35.877826 2584 patch_prober.go:28] interesting pod/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 06 20:07:35.878365 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:35.877895 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 06 20:07:36.877490 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:36.877440 2584 patch_prober.go:28] interesting pod/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 06 20:07:36.877707 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:36.877520 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 06 20:07:37.371213 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.371174 2584 patch_prober.go:28] interesting pod/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:07:37.371606 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.371234 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:07:37.785641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.785617 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:37.881866 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.881839 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:37.882172 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.882144 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:37.882775 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.882758 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:37.889796 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889779 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.889873 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889810 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.889873 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889836 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhmdx\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.889954 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889870 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.889954 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889913 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.890050 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889952 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.890050 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.889998 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.890050 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890046 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.890198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890062 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:37.890198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890074 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket\") pod \"b9e97f40-d433-433f-b25e-30d530babc07\" (UID: \"b9e97f40-d433-433f-b25e-30d530babc07\") " Jul 06 20:07:37.890303 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890252 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:37.890303 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890280 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:37.890408 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890366 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:37.890494 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890476 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e97f40-d433-433f-b25e-30d530babc07-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.890533 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890501 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.890533 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890517 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.890533 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890530 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.890741 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.890716 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:37.892429 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.892399 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:37.892523 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.892482 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:37.892592 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.892567 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx" (OuterVolumeSpecName: "kube-api-access-vhmdx") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "kube-api-access-vhmdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:37.892648 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.892618 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9e97f40-d433-433f-b25e-30d530babc07" (UID: "b9e97f40-d433-433f-b25e-30d530babc07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:07:37.991634 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.991589 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhmdx\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-kube-api-access-vhmdx\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.991634 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.991625 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e97f40-d433-433f-b25e-30d530babc07-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.991634 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.991640 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e97f40-d433-433f-b25e-30d530babc07-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.991861 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.991652 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:37.991861 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:37.991668 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e97f40-d433-433f-b25e-30d530babc07-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:38.401079 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.400994 2584 generic.go:356] "Generic (PLEG): container finished" podID="b9e97f40-d433-433f-b25e-30d530babc07" containerID="c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3" exitCode=0 Jul 06 20:07:38.401079 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.401070 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" Jul 06 20:07:38.401569 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.401083 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerDied","Data":"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3"} Jul 06 20:07:38.401569 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.401151 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw" event={"ID":"b9e97f40-d433-433f-b25e-30d530babc07","Type":"ContainerDied","Data":"9e7a0d6b4df079e97a189d3065ff933cdbf4e083700483338e74c4f2d672f97a"} Jul 06 20:07:38.401569 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.401182 2584 scope.go:117] "RemoveContainer" containerID="c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3" Jul 06 20:07:38.411065 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.411042 2584 scope.go:117] "RemoveContainer" containerID="922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11" Jul 06 20:07:38.418682 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.418665 2584 scope.go:117] "RemoveContainer" containerID="c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3" Jul 06 20:07:38.418957 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:07:38.418917 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3\": container with ID starting with c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3 not found: ID does not exist" containerID="c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3" Jul 06 20:07:38.419056 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.418965 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3"} err="failed to get container status \"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3\": rpc error: code = NotFound desc = could not find container \"c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3\": container with ID starting with c4d20cab5d5f056cfd8011e2a5c9f683ecb43c0d742a42c341dbf3a9df1d13b3 not found: ID does not exist" Jul 06 20:07:38.419056 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.418988 2584 scope.go:117] "RemoveContainer" containerID="922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11" Jul 06 20:07:38.419258 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:07:38.419241 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11\": container with ID starting with 922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11 not found: ID does not exist" containerID="922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11" Jul 06 20:07:38.419311 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.419272 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11"} err="failed to get container status \"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11\": rpc error: code = NotFound desc = could not find container \"922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11\": container with ID starting with 922244a6813ef265cef2bba00f13fccf7a2bf376ee16936b4f7c9c73ed88dc11 not found: ID does not exist" Jul 06 20:07:38.431823 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.431801 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:38.440720 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:38.440700 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e125-openshift-default-5bff85b797-gszcw"] Jul 06 20:07:39.223710 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:39.223680 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e97f40-d433-433f-b25e-30d530babc07" path="/var/lib/kubelet/pods/b9e97f40-d433-433f-b25e-30d530babc07/volumes" Jul 06 20:07:39.955003 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:39.954971 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:40.110373 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.110339 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:40.270590 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.270499 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:40.285219 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.285176 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:40.411883 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.411846 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" containerID="cri-o://9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec" gracePeriod=30 Jul 06 20:07:40.479171 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.479139 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:40.834005 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:40.833970 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:41.165499 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165467 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165884 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" containerName="maas-api" Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165898 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" containerName="maas-api" Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165910 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165916 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165963 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.166010 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.165970 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.166318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.166033 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2806cd-5492-44b6-bf23-0b7836dc29bc" containerName="maas-api" Jul 06 20:07:41.166318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.166041 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.166318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.166048 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e97f40-d433-433f-b25e-30d530babc07" containerName="istio-proxy" Jul 06 20:07:41.168832 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.168810 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.171616 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.171595 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ad5c84-openshift-default-dockercfg-45zdw\"" Jul 06 20:07:41.183647 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.183625 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:07:41.219038 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.219007 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:41.322951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.322890 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.322951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.322947 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzt4q\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.322967 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323072 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323115 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323297 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323221 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323297 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323265 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323399 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323386 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.323464 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.323403 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.424785 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424695 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.424785 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424732 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.424785 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.424785 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424848 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzt4q\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424865 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424921 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.424990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425169 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.425159 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425449 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.425240 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425449 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.425274 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425519 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.425463 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.425702 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.425684 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.427196 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.427166 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.427397 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.427378 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.439184 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.439162 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.440695 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.440671 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzt4q\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q\") pod \"e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.479471 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.479438 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:41.523185 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.523132 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:41.621190 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.621163 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:07:41.622909 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:07:41.622885 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058feb60_3c4a_4e2f_8425_9aa2a403dd87.slice/crio-95611bc96c4fceacadd4b87ca42232e47998a186a10d55a2488bdfc5c316e68e WatchSource:0}: Error finding container 95611bc96c4fceacadd4b87ca42232e47998a186a10d55a2488bdfc5c316e68e: Status 404 returned error can't find the container with id 95611bc96c4fceacadd4b87ca42232e47998a186a10d55a2488bdfc5c316e68e Jul 06 20:07:41.625041 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.625013 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:41.625121 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.625072 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:41.625121 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:41.625099 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:42.423115 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:42.423080 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerStarted","Data":"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3"} Jul 06 20:07:42.423509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:42.423132 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerStarted","Data":"95611bc96c4fceacadd4b87ca42232e47998a186a10d55a2488bdfc5c316e68e"} Jul 06 20:07:42.449833 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:42.449784 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podStartSLOduration=1.449768671 podStartE2EDuration="1.449768671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:42.447399228 +0000 UTC m=+2097.795042405" watchObservedRunningTime="2026-07-06 20:07:42.449768671 +0000 UTC m=+2097.797411846" Jul 06 20:07:42.479965 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:42.479913 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:42.485374 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:42.485348 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:43.428695 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:43.428665 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:43.429535 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:43.429515 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:07:44.415175 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:44.415144 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:44.564052 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:44.564018 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:44.673642 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:44.673563 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:44.776866 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:44.776837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:45.264904 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.264871 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:45.372574 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.372497 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:45.676541 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.676519 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:45.769429 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769393 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769605 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769455 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769605 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769490 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769605 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769531 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769605 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769572 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769612 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769636 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769772 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.769812 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769807 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xbkn\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn\") pod \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\" (UID: \"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62\") " Jul 06 20:07:45.770145 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.769801 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:45.770229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.770009 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:45.770229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.770044 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:45.770329 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.770254 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:45.770418 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.770394 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:45.771873 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.771848 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:07:45.772105 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.772082 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:45.772172 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.772130 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn" (OuterVolumeSpecName: "kube-api-access-8xbkn") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "kube-api-access-8xbkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:45.772427 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.772406 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" (UID: "a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870810 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870837 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870846 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870854 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbkn\" (UniqueName: \"kubernetes.io/projected/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-kube-api-access-8xbkn\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870864 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870881 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870889 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.870890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870897 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:45.871264 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:45.870905 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:07:46.446830 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.446794 2584 generic.go:356] "Generic (PLEG): container finished" podID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerID="9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec" exitCode=0 Jul 06 20:07:46.447059 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.446844 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" Jul 06 20:07:46.447059 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.446871 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerDied","Data":"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec"} Jul 06 20:07:46.447059 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.446952 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt" event={"ID":"a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62","Type":"ContainerDied","Data":"9c54b36ecc2df150ed5a00aac334efa72557e083df9923117f5267a6f38afc73"} Jul 06 20:07:46.447059 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.446984 2584 scope.go:117] "RemoveContainer" containerID="9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec" Jul 06 20:07:46.457075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.457058 2584 scope.go:117] "RemoveContainer" containerID="db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e" Jul 06 20:07:46.464864 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.464847 2584 scope.go:117] "RemoveContainer" containerID="9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec" Jul 06 20:07:46.465275 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:07:46.465256 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec\": container with ID starting with 9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec not found: ID does not exist" containerID="9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec" Jul 06 20:07:46.465351 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.465283 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec"} err="failed to get container status \"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec\": rpc error: code = NotFound desc = could not find container \"9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec\": container with ID starting with 9784e470bafb8653d9945bcf6b880dd3834a2b9726df380fc5da75c8cf6e42ec not found: ID does not exist" Jul 06 20:07:46.465351 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.465300 2584 scope.go:117] "RemoveContainer" containerID="db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e" Jul 06 20:07:46.465595 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:07:46.465576 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e\": container with ID starting with db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e not found: ID does not exist" containerID="db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e" Jul 06 20:07:46.465641 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.465600 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e"} err="failed to get container status \"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e\": rpc error: code = NotFound desc = could not find container \"db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e\": container with ID starting with db978edcdb2929d90b0738f4c520b94363d240b32341bc786ba6f1ab4f41075e not found: ID does not exist" Jul 06 20:07:46.480947 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.480910 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:46.486834 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:46.486812 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-618838-openshift-default-7c894c8b7c-4kcvt"] Jul 06 20:07:47.223710 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:47.223677 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" path="/var/lib/kubelet/pods/a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62/volumes" Jul 06 20:07:48.924846 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:48.924814 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:49.280426 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.280340 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:49.569104 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569024 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:07:49.569430 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569417 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:07:49.569495 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569431 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:07:49.569495 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569452 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:07:49.569495 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569457 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:07:49.569602 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.569514 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:07:49.571430 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.571414 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.574427 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.574407 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d1d0db-openshift-default-dockercfg-k57qc\"" Jul 06 20:07:49.589163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.589136 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:07:49.605543 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605506 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605543 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605537 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605587 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605643 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605663 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7kh\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.605777 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605742 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.606069 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605798 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.606069 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.605823 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.610751 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.610702 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:49.706522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706482 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.706668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706529 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.706668 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706581 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7kh\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.706841 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706818 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.706940 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706915 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707003 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.706971 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707062 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707039 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707112 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707067 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707111 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707213 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707151 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707263 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707235 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707443 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707419 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707613 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707588 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.707823 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.707804 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.708692 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.708671 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.709460 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.709443 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.720341 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.720309 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.720477 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.720354 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7kh\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh\") pod \"e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:49.795406 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.795373 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:49.882230 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:49.882129 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:50.034671 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.034642 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:07:50.037097 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:07:50.037063 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ee79a1_5aa0_4d18_8dda_5c07ca621cc1.slice/crio-9c897c4eb56e244c666b8f6ab3e6345246d96d07b333f9db54430bef1731b2b0 WatchSource:0}: Error finding container 9c897c4eb56e244c666b8f6ab3e6345246d96d07b333f9db54430bef1731b2b0: Status 404 returned error can't find the container with id 9c897c4eb56e244c666b8f6ab3e6345246d96d07b333f9db54430bef1731b2b0 Jul 06 20:07:50.039441 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.039404 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:50.039530 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.039473 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:50.039530 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.039510 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:07:50.470546 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.470505 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerStarted","Data":"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31"} Jul 06 20:07:50.470728 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.470563 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerStarted","Data":"9c897c4eb56e244c666b8f6ab3e6345246d96d07b333f9db54430bef1731b2b0"} Jul 06 20:07:50.496234 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.496165 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podStartSLOduration=1.496146186 podStartE2EDuration="1.496146186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:50.494266737 +0000 UTC m=+2105.841909913" watchObservedRunningTime="2026-07-06 20:07:50.496146186 +0000 UTC m=+2105.843789361" Jul 06 20:07:50.882409 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:50.882325 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:51.887451 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:51.887420 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:52.482200 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:52.482167 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:52.483114 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:52.483093 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:07:52.587951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:52.583083 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:52.680133 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:52.680087 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:52.895046 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:52.894956 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:53.107673 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:53.107640 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:53.589773 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:53.589731 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:53.675573 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:53.675506 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:56.987861 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:56.987817 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:59.471106 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:59.471059 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:07:59.471572 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:07:59.471132 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:08:00.535834 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.535799 2584 generic.go:356] "Generic (PLEG): container finished" podID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerID="4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3" exitCode=0 Jul 06 20:08:00.536349 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.535869 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerDied","Data":"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3"} Jul 06 20:08:00.536349 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.536308 2584 scope.go:117] "RemoveContainer" containerID="4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3" Jul 06 20:08:00.537336 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.537311 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:00.537400 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.537377 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:00.537436 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:00.537408 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:01.543296 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:01.543266 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerStarted","Data":"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c"} Jul 06 20:08:02.480636 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:02.480603 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:08:02.481143 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:02.481114 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 20:08:02.481284 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:02.481188 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 20:08:03.480742 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:03.480703 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 20:08:03.481359 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:03.481323 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 20:08:04.480810 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:04.480764 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 20:08:04.481461 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:04.481426 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 20:08:05.480474 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:05.480429 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 20:08:05.480677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:05.480504 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 20:08:06.484800 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:06.484763 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:08:06.485480 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:06.485460 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:08:06.486298 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:06.486272 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:08:06.815495 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:06.815403 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:06.915306 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:06.915264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:07.483922 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.483845 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 06 20:08:07.484134 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.483914 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 06 20:08:07.581871 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.581787 2584 generic.go:356] "Generic (PLEG): container finished" podID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerID="51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31" exitCode=0 Jul 06 20:08:07.582269 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.581869 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerDied","Data":"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31"} Jul 06 20:08:07.582406 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.582390 2584 scope.go:117] "RemoveContainer" containerID="51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31" Jul 06 20:08:07.583369 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.583339 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:07.583437 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.583421 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:07.583488 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:07.583473 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 20:08:08.591183 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:08.591150 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerStarted","Data":"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3"} Jul 06 20:08:08.882749 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:08.882638 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:08:08.883125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:08.883090 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 06 20:08:08.883248 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:08.883166 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 06 20:08:09.883458 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:09.883423 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 06 20:08:09.884460 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:09.884057 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 06 20:08:10.883209 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:10.882700 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 06 20:08:10.883209 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:10.882782 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 06 20:08:11.882630 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:11.882590 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 06 20:08:11.883276 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:11.883237 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 06 20:08:12.886195 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:12.886158 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:08:12.886670 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:12.886536 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:08:12.887097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:12.887077 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:08:13.596466 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:13.596437 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:13.675236 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:13.675209 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:15.143870 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:15.143842 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:16.482135 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:16.482100 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:16.965131 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:16.965085 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:35.522256 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:35.522226 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:43.086989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:43.086895 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7867bf745b-6299t" podUID="62cce9ad-9dbc-43cc-9df4-0bb84d980a68" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:08:43.815354 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:43.815315 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:43.916318 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:08:43.916272 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.745707 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:48.745664 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:49.745680 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:49.745647 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:57.916967 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.916912 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:10:57.917431 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.917417 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f65f70-d98c-4a6a-b9c3-26fa23a9fb62" containerName="istio-proxy" Jul 06 20:10:57.920595 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.920572 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.923431 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.923407 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ad5c84\"/\"openshift-service-ca.crt\"" Jul 06 20:10:57.923552 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.923411 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ad5c84\"/\"default-dockercfg-4w4g5\"" Jul 06 20:10:57.923552 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.923470 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ad5c84\"/\"sub-test-model-ad5c84-kserve-self-signed-certs\"" Jul 06 20:10:57.923552 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.923513 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ad5c84\"/\"kube-root-ca.crt\"" Jul 06 20:10:57.932517 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.932496 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:10:57.954941 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.954904 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:57.977655 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.977624 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.977811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.977680 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.977811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.977735 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrpz\" (UniqueName: \"kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.977811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.977756 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.978024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.977919 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:57.978135 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:57.978037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079180 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079253 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpz\" (UniqueName: \"kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079280 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079352 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079387 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079652 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079451 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079741 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079714 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079811 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079756 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.079968 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.079892 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.081610 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.081589 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.081845 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.081830 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.089421 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.089402 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrpz\" (UniqueName: \"kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz\") pod \"sub-test-model-ad5c84-kserve-df55c9797-5sqvk\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.230279 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.230181 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:10:58.365295 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.365255 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:10:58.367540 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:10:58.367512 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0131db2_b415_4b10_9209_26322308c9a4.slice/crio-c4ac829d3a7330f60461bdadb3ed54047ce57213e1e686d8e6a993d08291d338 WatchSource:0}: Error finding container c4ac829d3a7330f60461bdadb3ed54047ce57213e1e686d8e6a993d08291d338: Status 404 returned error can't find the container with id c4ac829d3a7330f60461bdadb3ed54047ce57213e1e686d8e6a993d08291d338 Jul 06 20:10:58.443223 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.443185 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" event={"ID":"b0131db2-b415-4b10-9209-26322308c9a4","Type":"ContainerStarted","Data":"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca"} Jul 06 20:10:58.443390 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.443235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" event={"ID":"b0131db2-b415-4b10-9209-26322308c9a4","Type":"ContainerStarted","Data":"c4ac829d3a7330f60461bdadb3ed54047ce57213e1e686d8e6a993d08291d338"} Jul 06 20:10:58.459539 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:10:58.459489 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" podStartSLOduration=1.459473599 podStartE2EDuration="1.459473599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:10:58.458385413 +0000 UTC m=+2293.806028592" watchObservedRunningTime="2026-07-06 20:10:58.459473599 +0000 UTC m=+2293.807116774" Jul 06 20:11:08.231211 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:11:08.231178 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:11:08.231618 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:11:08.231247 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:11:08.243905 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:11:08.243877 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:11:08.516990 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:11:08.516897 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:11:14.326999 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:11:14.326964 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:03.563392 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:03.563355 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:12:03.585799 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:03.585773 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:04.521338 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:04.521303 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:14.807278 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:14.807223 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:18.585053 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.584990 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" podUID="b0131db2-b415-4b10-9209-26322308c9a4" containerName="main" containerID="cri-o://18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca" gracePeriod=45 Jul 06 20:12:18.839436 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.839378 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:12:18.858213 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.858178 2584 generic.go:356] "Generic (PLEG): container finished" podID="b0131db2-b415-4b10-9209-26322308c9a4" containerID="18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca" exitCode=0 Jul 06 20:12:18.858380 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.858223 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" Jul 06 20:12:18.858380 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.858268 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" event={"ID":"b0131db2-b415-4b10-9209-26322308c9a4","Type":"ContainerDied","Data":"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca"} Jul 06 20:12:18.858380 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.858309 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk" event={"ID":"b0131db2-b415-4b10-9209-26322308c9a4","Type":"ContainerDied","Data":"c4ac829d3a7330f60461bdadb3ed54047ce57213e1e686d8e6a993d08291d338"} Jul 06 20:12:18.858380 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.858332 2584 scope.go:117] "RemoveContainer" containerID="18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca" Jul 06 20:12:18.868724 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.868698 2584 scope.go:117] "RemoveContainer" containerID="18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca" Jul 06 20:12:18.869081 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:12:18.869055 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca\": container with ID starting with 18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca not found: ID does not exist" containerID="18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca" Jul 06 20:12:18.869194 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.869095 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca"} err="failed to get container status \"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca\": rpc error: code = NotFound desc = could not find container \"18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca\": container with ID starting with 18df429f6216a3829d8057899e2ef42a19bd57b8ddd70fb2b8bcd4b945ccf9ca not found: ID does not exist" Jul 06 20:12:18.875440 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875407 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfrpz\" (UniqueName: \"kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875462 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875496 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875570 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875539 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875726 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875628 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875726 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875670 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home\") pod \"b0131db2-b415-4b10-9209-26322308c9a4\" (UID: \"b0131db2-b415-4b10-9209-26322308c9a4\") " Jul 06 20:12:18.875836 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875734 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:12:18.875836 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.875748 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:12:18.876081 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.876057 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home" (OuterVolumeSpecName: "home") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:12:18.876161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.876082 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-model-cache\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:18.876161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.876123 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-tmp-dir\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:18.877775 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.877753 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz" (OuterVolumeSpecName: "kube-api-access-rfrpz") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "kube-api-access-rfrpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:12:18.878067 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.878039 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:12:18.878067 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.878043 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm" (OuterVolumeSpecName: "dshm") pod "b0131db2-b415-4b10-9209-26322308c9a4" (UID: "b0131db2-b415-4b10-9209-26322308c9a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:12:18.976678 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.976637 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-dshm\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:18.976678 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.976672 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0131db2-b415-4b10-9209-26322308c9a4-home\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:18.976678 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.976685 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrpz\" (UniqueName: \"kubernetes.io/projected/b0131db2-b415-4b10-9209-26322308c9a4-kube-api-access-rfrpz\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:18.976918 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:18.976699 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0131db2-b415-4b10-9209-26322308c9a4-tls-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:12:19.183150 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:19.183117 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:12:19.189607 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:19.189583 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ad5c84/sub-test-model-ad5c84-kserve-df55c9797-5sqvk"] Jul 06 20:12:19.223991 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:19.223956 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0131db2-b415-4b10-9209-26322308c9a4" path="/var/lib/kubelet/pods/b0131db2-b415-4b10-9209-26322308c9a4/volumes" Jul 06 20:12:24.938209 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:24.938159 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:25.039205 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:25.039171 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:38.288556 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.288526 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:12:38.289024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.288901 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0131db2-b415-4b10-9209-26322308c9a4" containerName="main" Jul 06 20:12:38.289024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.288912 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0131db2-b415-4b10-9209-26322308c9a4" containerName="main" Jul 06 20:12:38.289024 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.289001 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0131db2-b415-4b10-9209-26322308c9a4" containerName="main" Jul 06 20:12:38.292052 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.292033 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.294774 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.294756 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d1d0db\"/\"openshift-service-ca.crt\"" Jul 06 20:12:38.294890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.294860 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d1d0db\"/\"default-dockercfg-cjs8d\"" Jul 06 20:12:38.294890 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.294859 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d1d0db\"/\"rate-test-model-d1d0db-kserve-self-signed-certs\"" Jul 06 20:12:38.295044 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.294987 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d1d0db\"/\"kube-root-ca.crt\"" Jul 06 20:12:38.314161 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.314135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:38.339667 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.339636 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:12:38.477699 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477669 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.477906 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477712 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.477906 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477745 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.477906 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477847 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.478076 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477920 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.478076 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.477994 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kccm\" (UniqueName: \"kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578546 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578546 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578501 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kccm\" (UniqueName: \"kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578735 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578624 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578735 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578686 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578742 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578815 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578798 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.578952 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.578900 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.579105 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.579088 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.579190 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.579167 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.580920 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.580898 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.581260 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.581242 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.590102 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.590075 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kccm\" (UniqueName: \"kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm\") pod \"rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.601619 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.601596 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:38.732833 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.732800 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:12:38.735693 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:12:38.735665 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecd1422a_9c8f_4a01_9273_d0b5f430ebb8.slice/crio-8da6ababb8ea8c242fbe29c1c530de0daa0ce7253ec088256123abbbb74c45b2 WatchSource:0}: Error finding container 8da6ababb8ea8c242fbe29c1c530de0daa0ce7253ec088256123abbbb74c45b2: Status 404 returned error can't find the container with id 8da6ababb8ea8c242fbe29c1c530de0daa0ce7253ec088256123abbbb74c45b2 Jul 06 20:12:38.960237 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.960197 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" event={"ID":"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8","Type":"ContainerStarted","Data":"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2"} Jul 06 20:12:38.960410 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.960247 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" event={"ID":"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8","Type":"ContainerStarted","Data":"8da6ababb8ea8c242fbe29c1c530de0daa0ce7253ec088256123abbbb74c45b2"} Jul 06 20:12:38.976698 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:38.976385 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" podStartSLOduration=0.976365121 podStartE2EDuration="976.365121ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:12:38.97610983 +0000 UTC m=+2394.323753005" watchObservedRunningTime="2026-07-06 20:12:38.976365121 +0000 UTC m=+2394.324008297" Jul 06 20:12:43.885031 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:43.884996 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:48.605625 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:48.605589 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:48.606027 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:48.605651 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:48.617990 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:48.617968 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:12:49.021564 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:12:49.021537 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:13:08.496262 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:08.496217 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:08.611054 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:08.611017 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:16.661547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:16.661513 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:17.806841 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:17.806809 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:22.994227 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:22.994189 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:23.982960 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:23.982905 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:13:24.004408 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:24.004378 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:25.037017 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:25.036904 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:25.478658 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:25.478619 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:25.615731 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:25.615701 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:25.715283 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:25.715249 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:25.958131 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:25.958098 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:30.544027 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:30.543994 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:39.002724 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.002649 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerName="main" containerID="cri-o://ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2" gracePeriod=45 Jul 06 20:13:39.010978 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.010946 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Jul 06 20:13:39.261958 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.261877 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:13:39.271297 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.271268 2584 generic.go:356] "Generic (PLEG): container finished" podID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerID="ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2" exitCode=0 Jul 06 20:13:39.271435 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.271303 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" Jul 06 20:13:39.271435 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.271354 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" event={"ID":"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8","Type":"ContainerDied","Data":"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2"} Jul 06 20:13:39.271435 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.271394 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8" event={"ID":"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8","Type":"ContainerDied","Data":"8da6ababb8ea8c242fbe29c1c530de0daa0ce7253ec088256123abbbb74c45b2"} Jul 06 20:13:39.271435 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.271412 2584 scope.go:117] "RemoveContainer" containerID="ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2" Jul 06 20:13:39.280223 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.279869 2584 scope.go:117] "RemoveContainer" containerID="ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2" Jul 06 20:13:39.280444 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:13:39.280412 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2\": container with ID starting with ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2 not found: ID does not exist" containerID="ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2" Jul 06 20:13:39.280601 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.280456 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2"} err="failed to get container status \"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2\": rpc error: code = NotFound desc = could not find container \"ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2\": container with ID starting with ed0bd4c47249bb1e02feddd7ee42ebd0d06c448b38c5896373ea495898a0e2c2 not found: ID does not exist" Jul 06 20:13:39.345942 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.345908 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.345988 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346030 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346050 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346069 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kccm\" (UniqueName: \"kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346125 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346095 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir\") pod \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\" (UID: \"ecd1422a-9c8f-4a01-9273-d0b5f430ebb8\") " Jul 06 20:13:39.346396 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346219 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:39.346396 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346314 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home" (OuterVolumeSpecName: "home") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:39.346512 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346412 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:39.346568 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346549 2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tmp-dir\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.346618 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346570 2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-model-cache\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.346618 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.346583 2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-home\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.348347 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.348319 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm" (OuterVolumeSpecName: "dshm") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:39.348461 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.348386 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:13:39.348461 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.348391 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm" (OuterVolumeSpecName: "kube-api-access-2kccm") pod "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" (UID: "ecd1422a-9c8f-4a01-9273-d0b5f430ebb8"). InnerVolumeSpecName "kube-api-access-2kccm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:13:39.448020 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.447990 2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-tls-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.448020 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.448017 2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-dshm\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.448020 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.448026 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kccm\" (UniqueName: \"kubernetes.io/projected/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8-kube-api-access-2kccm\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:39.593810 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.593777 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:13:39.597697 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:39.597675 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d1d0db/rate-test-model-d1d0db-kserve-f8ccbdd99-5kvw8"] Jul 06 20:13:41.223221 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:41.223193 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" path="/var/lib/kubelet/pods/ecd1422a-9c8f-4a01-9273-d0b5f430ebb8/volumes" Jul 06 20:13:54.188188 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.188152 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:54.447378 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.447282 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:13:54.447549 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.447515 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" containerID="cri-o://d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c" gracePeriod=30 Jul 06 20:13:54.465256 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.465226 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:54.715555 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.715469 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:54.912670 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:54.912632 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:55.281908 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:55.281871 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:55.440438 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:55.440394 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:55.536695 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:55.536611 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:55.788131 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:55.788046 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:59.705330 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.705290 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:13:59.853198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853102 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzt4q\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853140 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853198 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853177 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853228 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853260 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853295 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853324 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853376 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853522 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853405 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs\") pod \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\" (UID: \"058feb60-3c4a-4e2f-8425-9aa2a403dd87\") " Jul 06 20:13:59.853813 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853531 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:59.853813 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853583 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:59.853813 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853615 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:13:59.853989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853818 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:59.853989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853873 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.853989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853894 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.853989 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.853910 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.854667 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.854644 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data" (OuterVolumeSpecName: "istio-data") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:59.855452 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.855425 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:13:59.855592 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.855575 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:13:59.855849 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.855819 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q" (OuterVolumeSpecName: "kube-api-access-bzt4q") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "kube-api-access-bzt4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:13:59.855966 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.855852 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token" (OuterVolumeSpecName: "istio-token") pod "058feb60-3c4a-4e2f-8425-9aa2a403dd87" (UID: "058feb60-3c4a-4e2f-8425-9aa2a403dd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:13:59.955418 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955379 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.955418 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955409 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.955418 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955419 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058feb60-3c4a-4e2f-8425-9aa2a403dd87-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.955418 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955427 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzt4q\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-kube-api-access-bzt4q\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.955691 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955437 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:13:59.955691 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:13:59.955445 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058feb60-3c4a-4e2f-8425-9aa2a403dd87-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:00.378385 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.378351 2584 generic.go:356] "Generic (PLEG): container finished" podID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerID="d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c" exitCode=0 Jul 06 20:14:00.378547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.378402 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" Jul 06 20:14:00.378547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.378399 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerDied","Data":"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c"} Jul 06 20:14:00.378547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.378515 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9" event={"ID":"058feb60-3c4a-4e2f-8425-9aa2a403dd87","Type":"ContainerDied","Data":"95611bc96c4fceacadd4b87ca42232e47998a186a10d55a2488bdfc5c316e68e"} Jul 06 20:14:00.378547 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.378542 2584 scope.go:117] "RemoveContainer" containerID="d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c" Jul 06 20:14:00.387786 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.387766 2584 scope.go:117] "RemoveContainer" containerID="4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3" Jul 06 20:14:00.395399 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.395377 2584 scope.go:117] "RemoveContainer" containerID="d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c" Jul 06 20:14:00.395627 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:14:00.395609 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c\": container with ID starting with d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c not found: ID does not exist" containerID="d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c" Jul 06 20:14:00.395677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.395636 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c"} err="failed to get container status \"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c\": rpc error: code = NotFound desc = could not find container \"d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c\": container with ID starting with d14398c1cdb6e26cef0e59b112d847ae8e6c6da07c6a106160c5439aaf92ae0c not found: ID does not exist" Jul 06 20:14:00.395677 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.395653 2584 scope.go:117] "RemoveContainer" containerID="4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3" Jul 06 20:14:00.395857 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:14:00.395839 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3\": container with ID starting with 4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3 not found: ID does not exist" containerID="4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3" Jul 06 20:14:00.395898 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.395862 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3"} err="failed to get container status \"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3\": rpc error: code = NotFound desc = could not find container \"4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3\": container with ID starting with 4ad3679e63d3aa67a8806e89739874c33e4ed730453d31b17833bb8e703227c3 not found: ID does not exist" Jul 06 20:14:00.404321 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.404299 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:14:00.409829 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:00.409811 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ad5c84-openshift-default-8d884888c-5dxt9"] Jul 06 20:14:01.223259 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:01.223231 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" path="/var/lib/kubelet/pods/058feb60-3c4a-4e2f-8425-9aa2a403dd87/volumes" Jul 06 20:14:02.712157 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:02.712124 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:02.814491 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:02.814458 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:02.973881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:02.973799 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:14:02.974070 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:02.974049 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" containerID="cri-o://feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3" gracePeriod=30 Jul 06 20:14:02.996439 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:02.996404 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:03.221594 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:03.221563 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:08.229405 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.229380 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:14:08.327718 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327614 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c7kh\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.327718 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327679 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.327718 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327720 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327738 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327764 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327801 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327816 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327852 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328036 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.327915 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo\") pod \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\" (UID: \"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1\") " Jul 06 20:14:08.328291 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328077 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:14:08.328291 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328216 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:14:08.328291 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328264 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:14:08.328291 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328284 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-credential-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.328517 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328495 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:14:08.328949 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.328904 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:14:08.330100 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.330074 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh" (OuterVolumeSpecName: "kube-api-access-9c7kh") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "kube-api-access-9c7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:14:08.330197 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.330171 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:14:08.330261 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.330192 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:14:08.330261 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.330247 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" (UID: "e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:14:08.424094 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.424060 2584 generic.go:356] "Generic (PLEG): container finished" podID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerID="feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3" exitCode=0 Jul 06 20:14:08.424265 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.424109 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerDied","Data":"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3"} Jul 06 20:14:08.424265 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.424120 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" Jul 06 20:14:08.424265 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.424173 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk" event={"ID":"e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1","Type":"ContainerDied","Data":"9c897c4eb56e244c666b8f6ab3e6345246d96d07b333f9db54430bef1731b2b0"} Jul 06 20:14:08.424265 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.424195 2584 scope.go:117] "RemoveContainer" containerID="feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3" Jul 06 20:14:08.428975 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.428913 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istiod-ca-cert\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.428975 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.428976 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-certs\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.428975 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.428985 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-envoy\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.429163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.428993 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-token\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.429163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.429001 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-workload-socket\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.429163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.429009 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-podinfo\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.429163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.429017 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7kh\" (UniqueName: \"kubernetes.io/projected/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-kube-api-access-9c7kh\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.429163 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.429026 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1-istio-data\") on node \"ip-10-0-136-184.ec2.internal\" DevicePath \"\"" Jul 06 20:14:08.433771 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.433754 2584 scope.go:117] "RemoveContainer" containerID="51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31" Jul 06 20:14:08.441997 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.441977 2584 scope.go:117] "RemoveContainer" containerID="feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3" Jul 06 20:14:08.442264 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:14:08.442246 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3\": container with ID starting with feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3 not found: ID does not exist" containerID="feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3" Jul 06 20:14:08.442328 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.442274 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3"} err="failed to get container status \"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3\": rpc error: code = NotFound desc = could not find container \"feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3\": container with ID starting with feb5eedd13568b1f10ce3e069cc45261a08e4cc9019c71816b58d66ab85326f3 not found: ID does not exist" Jul 06 20:14:08.442328 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.442292 2584 scope.go:117] "RemoveContainer" containerID="51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31" Jul 06 20:14:08.442537 ip-10-0-136-184 kubenswrapper[2584]: E0706 20:14:08.442521 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31\": container with ID starting with 51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31 not found: ID does not exist" containerID="51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31" Jul 06 20:14:08.442579 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.442542 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31"} err="failed to get container status \"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31\": rpc error: code = NotFound desc = could not find container \"51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31\": container with ID starting with 51eda2e117e2e584c653b3d5c07cc63f98430e222920e8fe6727174df5e7ef31 not found: ID does not exist" Jul 06 20:14:08.446165 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.446145 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:14:08.450039 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.450016 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d1d0db-openshift-default-6cd97c8cbc-h6frk"] Jul 06 20:14:08.499082 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.499060 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpjb9_2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3/manager/1.log" Jul 06 20:14:08.789093 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.789065 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-554c6857df-kg4lf_11b52f75-be8b-4508-86dd-5e53e559aac2/maas-api/0.log" Jul 06 20:14:08.905606 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:08.905572 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-5sdc8_3eb94900-ecce-4cc9-9fd7-69bdf040000d/cleanup/0.log" Jul 06 20:14:09.028695 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.028661 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-pvrhh_a7fc445c-d471-4536-9ca8-0db8d970d1d3/cleanup/0.log" Jul 06 20:14:09.154685 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.154610 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7867bf745b-6299t_62cce9ad-9dbc-43cc-9df4-0bb84d980a68/manager/0.log" Jul 06 20:14:09.223145 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.223113 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" path="/var/lib/kubelet/pods/e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1/volumes" Jul 06 20:14:09.519089 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.519061 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cbdc6848b-tb7x5_62d30bf8-8393-4f51-a745-66b1b35078bc/copy-manifests/0.log" Jul 06 20:14:09.527075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.527053 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cbdc6848b-tb7x5_62d30bf8-8393-4f51-a745-66b1b35078bc/manager/1.log" Jul 06 20:14:09.903589 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:09.903516 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-64rbd_25c99692-2841-4930-afa1-e8235f105604/postgres/0.log" Jul 06 20:14:11.540068 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:11.540037 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-llhq8_b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e/manager/1.log" Jul 06 20:14:11.773743 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:11.773711 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z8zcq_9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a/registry-server/0.log" Jul 06 20:14:11.902473 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:11.902395 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8fncd_497178e9-ba0b-4410-9f48-69e2b9d04673/manager/0.log" Jul 06 20:14:12.518460 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:12.518430 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bzclp_2780cfdc-2967-4e12-b65f-494ff2dc99d8/istio-proxy/0.log" Jul 06 20:14:13.001806 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:13.001777 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/1.log" Jul 06 20:14:13.119898 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:13.119868 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mw7np_4ededa9f-2440-478e-8ec5-4bf9292c2151/payload-pre-processing/0.log" Jul 06 20:14:13.254940 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:13.254855 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pmlkj_8fd2b566-4b66-4728-a953-df5b2acb65b8/payload-processing/0.log" Jul 06 20:14:13.728660 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:13.728630 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ff7km_5b2116eb-23f4-4d75-8127-7a8750bd4772/main/0.log" Jul 06 20:14:14.085886 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:14.085811 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnnkws_7db77586-64c4-4215-bbd4-c7b52fcaab84/main/0.log" Jul 06 20:14:18.155951 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:18.155902 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:18.224566 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:18.224525 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:21.386413 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:21.386369 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-89zgx_a6987347-09bc-4125-9ebc-00e37b431786/global-pull-secret-syncer/0.log" Jul 06 20:14:21.478474 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:21.478444 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" Jul 06 20:14:21.608486 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:21.608454 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-184.ec2.internal_a6c8ec2323b990d6957cedec6f5b7c41/haproxy/0.log" Jul 06 20:14:26.989638 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:26.989605 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-llhq8_b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e/manager/1.log" Jul 06 20:14:26.990070 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:26.989710 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-llhq8_b5f944e6-bd46-4b41-9e7c-9d7ba0dae93e/manager/0.log" Jul 06 20:14:27.060438 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:27.060409 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z8zcq_9e37dc3a-11e0-4261-a6a3-6cbce0bc2c4a/registry-server/0.log" Jul 06 20:14:27.191075 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:27.191049 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8fncd_497178e9-ba0b-4410-9f48-69e2b9d04673/manager/0.log" Jul 06 20:14:28.939899 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:28.939868 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/alertmanager/0.log" Jul 06 20:14:28.964187 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:28.964153 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/config-reloader/0.log" Jul 06 20:14:28.986207 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:28.986174 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/kube-rbac-proxy-web/0.log" Jul 06 20:14:29.008613 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.008586 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/kube-rbac-proxy/0.log" Jul 06 20:14:29.034331 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.034303 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/kube-rbac-proxy-metric/0.log" Jul 06 20:14:29.057396 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.057368 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/prom-label-proxy/0.log" Jul 06 20:14:29.080756 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.080731 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebea2914-f9b7-478e-b0bc-a35512ff3a89/init-config-reloader/0.log" Jul 06 20:14:29.248006 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.247905 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b8479df95-n4w9q_bb0b0248-7619-4236-a16d-573fad578c5b/metrics-server/0.log" Jul 06 20:14:29.330792 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.330750 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jwnb_e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3/node-exporter/0.log" Jul 06 20:14:29.358879 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.358855 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jwnb_e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3/kube-rbac-proxy/0.log" Jul 06 20:14:29.392689 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.392667 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8jwnb_e91e8398-7794-4b60-8a0c-d1fdcc7fa9a3/init-textfile/0.log" Jul 06 20:14:29.747945 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.747903 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/prometheus/0.log" Jul 06 20:14:29.788741 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.788706 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/config-reloader/0.log" Jul 06 20:14:29.814099 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.814061 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/thanos-sidecar/0.log" Jul 06 20:14:29.842613 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.842587 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/kube-rbac-proxy-web/0.log" Jul 06 20:14:29.868124 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.868093 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/kube-rbac-proxy/0.log" Jul 06 20:14:29.893648 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.893627 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/kube-rbac-proxy-thanos/0.log" Jul 06 20:14:29.917854 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:29.917800 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9fdeb009-3b9b-490f-b0a3-146aca6f31cc/init-config-reloader/0.log" Jul 06 20:14:30.011610 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.011533 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-lq5qj_408e8f38-c7b9-4ec4-af9a-45d0e4c1c919/prometheus-operator-admission-webhook/0.log" Jul 06 20:14:30.819679 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.819646 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d"] Jul 06 20:14:30.820043 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820031 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" Jul 06 20:14:30.820097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820044 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" Jul 06 20:14:30.820097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820052 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerName="main" Jul 06 20:14:30.820097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820057 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerName="main" Jul 06 20:14:30.820097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820074 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" Jul 06 20:14:30.820097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820079 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" Jul 06 20:14:30.820252 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820146 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" Jul 06 20:14:30.820252 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820154 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="058feb60-3c4a-4e2f-8425-9aa2a403dd87" containerName="istio-proxy" Jul 06 20:14:30.820252 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820161 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" Jul 06 20:14:30.820252 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820166 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ee79a1-5aa0-4d18-8dda-5c07ca621cc1" containerName="istio-proxy" Jul 06 20:14:30.820252 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.820174 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd1422a-9c8f-4a01-9273-d0b5f430ebb8" containerName="main" Jul 06 20:14:30.823332 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.823316 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:30.825537 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.825515 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcmql\"/\"kube-root-ca.crt\"" Jul 06 20:14:30.826081 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.826063 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kcmql\"/\"default-dockercfg-mh48k\"" Jul 06 20:14:30.826149 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.826078 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcmql\"/\"openshift-service-ca.crt\"" Jul 06 20:14:30.832509 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.832489 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d"] Jul 06 20:14:30.938229 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.938199 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ln5\" (UniqueName: \"kubernetes.io/projected/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-kube-api-access-79ln5\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:30.938413 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.938239 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-lib-modules\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:30.938413 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.938268 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-proc\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:30.938413 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.938305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-sys\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:30.938413 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:30.938329 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-podres\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039247 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039216 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ln5\" (UniqueName: \"kubernetes.io/projected/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-kube-api-access-79ln5\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039257 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-lib-modules\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039286 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-proc\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039328 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-sys\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039346 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-podres\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039419 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-proc\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039427 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-lib-modules\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039455 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-sys\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.039675 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.039471 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-podres\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.046656 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.046631 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ln5\" (UniqueName: \"kubernetes.io/projected/f1ab328e-2b65-4cfe-b253-68cad2f2a6ce-kube-api-access-79ln5\") pod \"perf-node-gather-daemonset-m2j2d\" (UID: \"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce\") " pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.133471 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.133380 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.267756 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.267728 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d"] Jul 06 20:14:31.270505 ip-10-0-136-184 kubenswrapper[2584]: W0706 20:14:31.270475 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1ab328e_2b65_4cfe_b253_68cad2f2a6ce.slice/crio-015f7a4e04e3e605d2ae16b3c88cafd9c54dd113a90fc1d3514cda7d95c3f06a WatchSource:0}: Error finding container 015f7a4e04e3e605d2ae16b3c88cafd9c54dd113a90fc1d3514cda7d95c3f06a: Status 404 returned error can't find the container with id 015f7a4e04e3e605d2ae16b3c88cafd9c54dd113a90fc1d3514cda7d95c3f06a Jul 06 20:14:31.537534 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.537498 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" event={"ID":"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce","Type":"ContainerStarted","Data":"3ec1304f70d9a3a55aa90a068d430b0a11892e2f75e728534e5c322002959b89"} Jul 06 20:14:31.537730 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.537561 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" event={"ID":"f1ab328e-2b65-4cfe-b253-68cad2f2a6ce","Type":"ContainerStarted","Data":"015f7a4e04e3e605d2ae16b3c88cafd9c54dd113a90fc1d3514cda7d95c3f06a"} Jul 06 20:14:31.537730 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.537589 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:31.559881 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:31.559832 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" podStartSLOduration=1.559817064 podStartE2EDuration="1.559817064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:31.558961323 +0000 UTC m=+2506.906604496" watchObservedRunningTime="2026-07-06 20:14:31.559817064 +0000 UTC m=+2506.907460238" Jul 06 20:14:34.260472 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:34.260440 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z7rf_56edb1f5-00f9-4330-b62e-de016f1387af/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:34.445612 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:34.445584 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9jm9_35a4fb1f-a335-4e23-9018-1fd5d559645a/dns/0.log" Jul 06 20:14:34.465434 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:34.465411 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9jm9_35a4fb1f-a335-4e23-9018-1fd5d559645a/kube-rbac-proxy/0.log" Jul 06 20:14:34.549919 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:34.549838 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qqkrj_03881413-57f5-4fc2-bc7d-94ae5db6c711/dns-node-resolver/0.log" Jul 06 20:14:35.171029 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:35.171002 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r2rtw_83256f13-fd24-4c93-938d-c5ee1e8ecc93/node-ca/0.log" Jul 06 20:14:36.115724 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:36.115687 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bzclp_2780cfdc-2967-4e12-b65f-494ff2dc99d8/istio-proxy/0.log" Jul 06 20:14:36.321073 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:36.321048 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/0.log" Jul 06 20:14:36.348693 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:36.348658 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/1.log" Jul 06 20:14:36.378326 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:36.378242 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mw7np_4ededa9f-2440-478e-8ec5-4bf9292c2151/payload-pre-processing/0.log" Jul 06 20:14:36.425797 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:36.425727 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pmlkj_8fd2b566-4b66-4728-a953-df5b2acb65b8/payload-processing/0.log" Jul 06 20:14:37.295051 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:37.295022 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzt8q_1bb27ca0-ec9c-486a-8e30-e6d07fac3183/serve-healthcheck-canary/0.log" Jul 06 20:14:37.552433 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:37.552369 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kcmql/perf-node-gather-daemonset-m2j2d" Jul 06 20:14:38.029302 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:38.029275 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbfds_29ddcb7d-7155-4fd8-9ae3-8c55069f30c0/kube-rbac-proxy/0.log" Jul 06 20:14:38.054242 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:38.054213 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbfds_29ddcb7d-7155-4fd8-9ae3-8c55069f30c0/exporter/0.log" Jul 06 20:14:38.077797 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:38.077764 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbfds_29ddcb7d-7155-4fd8-9ae3-8c55069f30c0/extractor/0.log" Jul 06 20:14:40.499026 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.498996 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpjb9_2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3/manager/0.log" Jul 06 20:14:40.499419 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.499046 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpjb9_2086bdc0-0aa4-4b9f-a1c9-a86274d6b4a3/manager/1.log" Jul 06 20:14:40.708582 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.708551 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-554c6857df-kg4lf_11b52f75-be8b-4508-86dd-5e53e559aac2/maas-api/0.log" Jul 06 20:14:40.730407 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.730383 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-5sdc8_3eb94900-ecce-4cc9-9fd7-69bdf040000d/cleanup/0.log" Jul 06 20:14:40.754780 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.754699 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-pvrhh_a7fc445c-d471-4536-9ca8-0db8d970d1d3/cleanup/0.log" Jul 06 20:14:40.855608 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.855579 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7867bf745b-6299t_62cce9ad-9dbc-43cc-9df4-0bb84d980a68/manager/0.log" Jul 06 20:14:40.986904 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:40.986876 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cbdc6848b-tb7x5_62d30bf8-8393-4f51-a745-66b1b35078bc/manager/1.log" Jul 06 20:14:41.000343 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.000320 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cbdc6848b-tb7x5_62d30bf8-8393-4f51-a745-66b1b35078bc/manager/0.log" Jul 06 20:14:41.016471 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.016398 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cbdc6848b-tb7x5_62d30bf8-8393-4f51-a745-66b1b35078bc/copy-manifests/0.log" Jul 06 20:14:41.065809 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.065781 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bzclp_2780cfdc-2967-4e12-b65f-494ff2dc99d8/istio-proxy/0.log" Jul 06 20:14:41.245950 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.245907 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-64rbd_25c99692-2841-4930-afa1-e8235f105604/postgres/0.log" Jul 06 20:14:41.284611 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.284520 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/0.log" Jul 06 20:14:41.311317 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.311289 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/1.log" Jul 06 20:14:41.332223 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.332192 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mw7np_4ededa9f-2440-478e-8ec5-4bf9292c2151/payload-pre-processing/0.log" Jul 06 20:14:41.372512 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:41.372483 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pmlkj_8fd2b566-4b66-4728-a953-df5b2acb65b8/payload-processing/0.log" Jul 06 20:14:42.488767 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.488737 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bzclp_2780cfdc-2967-4e12-b65f-494ff2dc99d8/istio-proxy/0.log" Jul 06 20:14:42.547097 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.547071 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59bc67679f-cw55b_cc40740a-3c88-4259-94f3-4bad03ac345d/manager/0.log" Jul 06 20:14:42.678385 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.678354 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/0.log" Jul 06 20:14:42.703596 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.703570 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gv6vf_9b11431b-349f-47d6-9a72-bbe310b8d5b3/istio-proxy/1.log" Jul 06 20:14:42.724040 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.724015 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-mw7np_4ededa9f-2440-478e-8ec5-4bf9292c2151/payload-pre-processing/0.log" Jul 06 20:14:42.771062 ip-10-0-136-184 kubenswrapper[2584]: I0706 20:14:42.770989 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pmlkj_8fd2b566-4b66-4728-a953-df5b2acb65b8/payload-processing/0.log"