Jul 27 21:52:37.687161 ip-10-0-134-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 21:52:37.687173 ip-10-0-134-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 21:52:37.687180 ip-10-0-134-139 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 21:52:37.687414 ip-10-0-134-139 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 21:52:47.753208 ip-10-0-134-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 27 21:52:47.753227 ip-10-0-134-139 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d9f3689e41ed465bab2ed85dca343af6 -- Jul 27 21:54:39.521450 ip-10-0-134-139 systemd[1]: Starting Kubernetes Kubelet... Jul 27 21:54:39.844943 ip-10-0-134-139 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:54:39.844943 ip-10-0-134-139 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 21:54:39.844943 ip-10-0-134-139 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:54:39.844943 ip-10-0-134-139 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 21:54:39.844943 ip-10-0-134-139 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:54:39.845728 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.845610 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 21:54:39.850837 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850821 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.850837 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850837 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850840 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850846 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850850 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850854 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850857 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850860 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850862 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850866 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850868 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850871 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850874 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850877 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850880 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850882 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850885 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850888 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850890 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850893 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850896 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.850911 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850899 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850903 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850905 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850908 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850910 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850913 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850916 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850918 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850926 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850929 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850932 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850934 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850936 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850939 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850942 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850944 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850947 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850949 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850951 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.851400 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850954 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850956 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850958 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850961 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850963 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850965 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850969 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850971 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850974 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850977 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850979 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850981 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850985 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850987 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850990 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850993 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850995 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.850998 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851001 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851003 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.851857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851006 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851008 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851010 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851013 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851015 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851018 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851020 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851022 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851025 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851027 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851029 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851032 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851034 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851038 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851043 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851047 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851050 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851053 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851057 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.852362 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851060 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851063 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851066 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851069 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851071 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851074 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851077 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851079 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851083 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851085 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851088 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851091 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851093 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851096 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851098 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851101 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851103 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851106 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851108 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851111 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.852955 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851841 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851849 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851852 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851856 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851858 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851861 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851864 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851867 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851869 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851872 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851874 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851877 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851879 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851883 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851885 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851888 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851890 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851893 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851895 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.853449 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851898 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851903 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851906 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851909 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851912 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851915 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851918 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851922 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851925 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851927 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851930 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851933 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851935 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851938 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851940 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851943 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851945 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851948 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851951 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.853913 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851953 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851956 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851958 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851961 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851964 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851967 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851969 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851972 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851974 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851977 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851980 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851983 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851985 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851988 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851991 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851994 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851997 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.851999 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852001 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852004 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.854390 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852006 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852009 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852011 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852013 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852016 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852018 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852021 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852023 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852025 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852028 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852030 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852032 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852035 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852037 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852040 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852042 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852045 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852047 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852049 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852052 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.854881 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852054 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852057 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852062 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852065 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852068 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852071 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852075 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852077 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852079 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852082 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852084 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852087 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852089 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852092 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852094 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852096 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852099 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852101 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852104 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.855397 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852106 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852109 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852491 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852496 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852499 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852502 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852505 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852509 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852511 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852515 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852518 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852520 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852523 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852526 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852529 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852532 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852534 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852537 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852539 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852542 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.855857 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852544 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852547 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852551 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852553 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852556 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852558 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852561 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852563 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852566 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852568 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852571 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852573 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852576 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852578 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852582 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852585 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852588 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852591 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852593 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.856349 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852596 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852599 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852602 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852604 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852608 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852611 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852614 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852617 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852620 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852623 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852625 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852628 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852630 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852633 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852635 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852637 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852640 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852642 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852645 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852647 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.856807 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852649 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852652 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852655 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852657 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852660 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852662 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852665 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852667 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852669 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852672 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852674 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852676 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852679 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852681 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852684 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852687 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852689 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852692 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852694 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852696 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.857311 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852699 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852702 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852704 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852707 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852709 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852711 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852714 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852716 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852719 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852721 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852723 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852726 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852728 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852731 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852733 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852735 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852737 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852741 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852745 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.857812 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852747 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852750 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.852753 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852821 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852828 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852836 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852845 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852851 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852856 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852863 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852869 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852872 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852875 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852878 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852881 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852885 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852888 2559 flags.go:64] FLAG: --cgroup-root="" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852891 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852894 2559 flags.go:64] FLAG: --client-ca-file="" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852896 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852899 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852903 2559 flags.go:64] FLAG: --cluster-domain="" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852906 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852909 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 21:54:39.858278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852914 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852917 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852921 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852924 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852927 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852930 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852934 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852936 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852939 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852943 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852945 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852949 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852952 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852955 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852959 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852962 2559 flags.go:64] FLAG: --enable-server="true" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852965 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852973 2559 flags.go:64] FLAG: --event-burst="100" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852976 2559 flags.go:64] FLAG: --event-qps="50" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852978 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852981 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852984 2559 flags.go:64] FLAG: --eviction-hard="" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852988 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852991 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852994 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 21:54:39.858841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.852998 2559 flags.go:64] FLAG: --eviction-soft="" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853000 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853003 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853006 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853008 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853011 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853014 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853018 2559 flags.go:64] FLAG: --feature-gates="" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853021 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853024 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853027 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853030 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853033 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853036 2559 flags.go:64] FLAG: --help="false" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853038 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-134-139.ec2.internal" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853041 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853044 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853047 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853050 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853054 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853056 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853060 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853063 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 21:54:39.859440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853066 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853069 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853072 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853075 2559 flags.go:64] FLAG: --kube-reserved="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853077 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853080 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853083 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853085 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853088 2559 flags.go:64] FLAG: --lock-file="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853092 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853095 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853097 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853103 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853105 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853108 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853111 2559 flags.go:64] FLAG: --logging-format="text" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853115 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853118 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853121 2559 flags.go:64] FLAG: --manifest-url="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853123 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853127 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853131 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853135 2559 flags.go:64] FLAG: --max-pods="110" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853138 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853140 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 21:54:39.860104 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853143 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853161 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853164 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853167 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853170 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853180 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853183 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853186 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853189 2559 flags.go:64] FLAG: --pod-cidr="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853192 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853197 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853200 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853203 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853206 2559 flags.go:64] FLAG: --port="10250" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853209 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853212 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf5c670c696a0a8a" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853215 2559 flags.go:64] FLAG: --qos-reserved="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853218 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853221 2559 flags.go:64] FLAG: --register-node="true" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853223 2559 flags.go:64] FLAG: --register-with-taints="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853227 2559 flags.go:64] FLAG: --registry-burst="10" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853229 2559 flags.go:64] FLAG: --registry-qps="5" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853232 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853236 2559 flags.go:64] FLAG: --reserved-memory="" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853243 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 21:54:39.860798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853245 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853248 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853251 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853254 2559 flags.go:64] FLAG: --runonce="false" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853257 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853260 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853263 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853266 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853268 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853271 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853274 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853277 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853280 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853284 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853287 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853289 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853292 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853295 2559 flags.go:64] FLAG: --system-cgroups="" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853297 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853303 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853306 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853308 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853312 2559 flags.go:64] FLAG: --tls-min-version="" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853315 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853317 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 21:54:39.861481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853321 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853324 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853326 2559 flags.go:64] FLAG: --v="2" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853331 2559 flags.go:64] FLAG: --version="false" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853334 2559 flags.go:64] FLAG: --vmodule="" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853340 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.853343 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853459 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853464 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853467 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853469 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853472 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853474 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853477 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853479 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853482 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853485 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853487 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853490 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853492 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853497 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.862103 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853499 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853502 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853504 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853507 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853509 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853512 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853514 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853516 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853519 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853521 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853524 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853526 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853529 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853532 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853535 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853538 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853542 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853546 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853550 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853553 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.862639 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853556 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853558 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853561 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853564 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853566 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853569 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853571 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853573 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853576 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853579 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853582 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853586 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853588 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853590 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853593 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853596 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853598 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853601 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853603 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853606 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.863111 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853608 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853611 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853613 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853616 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853618 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853621 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853624 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853626 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853630 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853632 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853635 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853637 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853640 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853642 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853645 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853647 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853650 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853652 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853655 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853657 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.863644 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853660 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853662 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853665 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853669 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853671 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853674 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853677 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853679 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853683 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853686 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853688 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853692 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853696 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853698 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853701 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853704 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853707 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853710 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853712 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.864196 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853715 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853719 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853722 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853724 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853727 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.853729 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.854417 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.861128 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.861142 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.861770 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.861780 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861845 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861849 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861852 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.864706 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861856 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861858 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861861 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861865 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861867 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861870 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861873 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861876 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861879 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861881 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861883 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861886 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861888 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861891 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861893 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861897 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861899 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861902 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861904 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861907 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.865053 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861910 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861914 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861918 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861921 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861924 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861927 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861930 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861933 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861936 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861939 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861941 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861944 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861947 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861949 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861952 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861955 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861958 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861961 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861964 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.865554 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861966 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861969 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861971 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861974 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861977 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861979 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861982 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861984 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861987 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861990 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861992 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861994 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861997 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.861999 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862002 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862004 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862006 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862009 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862012 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862014 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.866033 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862017 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862020 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862022 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862026 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862029 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862032 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862035 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862037 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862040 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862042 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862045 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862047 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862050 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862052 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862055 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862058 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862062 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862064 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862067 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.866540 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862070 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862073 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862076 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862078 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862081 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862084 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862086 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862089 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862091 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862093 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862096 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862098 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862101 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862103 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862106 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862109 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862112 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.866994 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862114 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.862119 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862258 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862264 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862267 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862271 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862273 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862277 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862280 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862282 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862285 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862289 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862292 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862295 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:54:39.867434 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862299 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862303 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862305 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862308 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862311 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862314 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862317 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862320 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862322 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862325 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862327 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862330 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862332 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862335 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862337 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862339 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862344 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862347 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862350 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:54:39.867790 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862352 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862355 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862357 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862360 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862362 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862365 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862367 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862370 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862372 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862375 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862377 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862379 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862382 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862384 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862386 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862389 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862391 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862394 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862396 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862398 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:54:39.868455 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862401 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862403 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862405 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862408 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862410 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862412 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862415 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862417 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862419 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862422 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862425 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862427 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862429 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862432 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862434 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862436 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862439 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862441 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862444 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862447 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:54:39.868952 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862449 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862451 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862454 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862456 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862459 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862461 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862464 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862466 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862469 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862471 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862474 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862476 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862478 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862481 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862483 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862485 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862488 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862492 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862495 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862497 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:54:39.869474 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862500 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862502 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862504 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862507 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862509 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862512 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862515 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:39.862517 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.862522 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.862635 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.864620 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.865990 2559 server.go:1019] "Starting client certificate rotation" Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.866091 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:54:39.869970 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.866122 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:54:39.870419 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.867894 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:54:39.870419 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.869416 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:54:39.878229 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.878210 2559 log.go:25] "Validated CRI v1 runtime API" Jul 27 21:54:39.880066 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.880050 2559 log.go:25] "Validated CRI v1 image API" Jul 27 21:54:39.881770 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.881753 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 21:54:39.884853 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.884823 2559 fs.go:135] Filesystem UUIDs: map[3dd86df4-e3c2-4750-a778-49b3894d2493:/dev/nvme0n1p4 5bef2562-483d-47a5-9fc4-beb0045535d4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 27 21:54:39.884937 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.884851 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 21:54:39.890078 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.889959 2559 manager.go:216] Machine: {Timestamp:2026-07-27 21:54:39.888776223 +0000 UTC m=+0.269848137 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099673 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec256eec689cdfaf71aba4150b933400 SystemUUID:ec256eec-689c-dfaf-71ab-a4150b933400 BootID:d9f3689e-41ed-465b-ab2e-d85dca343af6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:b2:cd:8c:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:b2:cd:8c:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:0d:81:f9:20:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 21:54:39.890078 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.890065 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 21:54:39.890258 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.890193 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 21:54:39.892669 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.892642 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 21:54:39.892801 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.892673 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 21:54:39.893276 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.893265 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 21:54:39.893309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.893281 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 21:54:39.893309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.893291 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:54:39.893373 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.893315 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:54:39.894106 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.894088 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:54:39.894215 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.894095 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 21:54:39.895649 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895637 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:54:39.895808 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895798 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 21:54:39.895847 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895829 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 27 21:54:39.895847 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895839 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 21:54:39.895995 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895853 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 21:54:39.895995 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895865 2559 kubelet.go:398] "Adding apiserver pod source" Jul 27 21:54:39.895995 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.895873 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 21:54:39.896244 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.896232 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:54:39.896288 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.896254 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:54:39.898017 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.897999 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 21:54:39.899677 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.899663 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 21:54:39.899722 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.899694 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 21:54:39.900947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900937 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 21:54:39.900981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900953 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 21:54:39.900981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900960 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 21:54:39.900981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900965 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 21:54:39.900981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900974 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 21:54:39.900981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900979 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900988 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.900994 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901001 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901006 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901019 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 21:54:39.901108 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901025 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 21:54:39.901576 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901566 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 21:54:39.901576 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.901576 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 21:54:39.904656 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.904643 2559 server.go:1268] "Started kubelet" Jul 27 21:54:39.904787 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.904739 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 21:54:39.904842 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.904812 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 21:54:39.904928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.904898 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 21:54:39.905544 ip-10-0-134-139 systemd[1]: Started Kubernetes Kubelet. Jul 27 21:54:39.906226 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.906210 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 21:54:39.906290 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.906206 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 27 21:54:39.909784 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.909753 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 21:54:39.909881 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.909804 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 21:54:39.910045 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.909984 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 21:54:39.912762 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.912743 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 21:54:39.914253 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.913414 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-139.ec2.internal.18c64431158cc070 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-139.ec2.internal,UID:ip-10-0-134-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-139.ec2.internal,},FirstTimestamp:2026-07-27 21:54:39.904612464 +0000 UTC m=+0.285684380,LastTimestamp:2026-07-27 21:54:39.904612464 +0000 UTC m=+0.285684380,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-139.ec2.internal,}" Jul 27 21:54:39.916462 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.916440 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 21:54:39.916462 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.916453 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 21:54:39.916831 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.916579 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:39.917006 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.916980 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 21:54:39.917006 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.916999 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 21:54:39.917129 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917079 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 21:54:39.917202 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917190 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 27 21:54:39.917275 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917213 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 27 21:54:39.917741 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917721 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 21:54:39.917741 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917741 2559 factory.go:55] Registering systemd factory Jul 27 21:54:39.917881 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917752 2559 factory.go:223] Registration of the systemd container factory successfully Jul 27 21:54:39.917998 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.917985 2559 factory.go:153] Registering CRI-O factory Jul 27 21:54:39.918038 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.918001 2559 factory.go:223] Registration of the crio container factory successfully Jul 27 21:54:39.918038 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.918022 2559 factory.go:103] Registering Raw factory Jul 27 21:54:39.918038 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.918035 2559 manager.go:1196] Started watching for new ooms in manager Jul 27 21:54:39.918635 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.918621 2559 manager.go:349] Starting recovery of all containers Jul 27 21:54:39.921401 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.921372 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 21:54:39.921522 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.921500 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 21:54:39.922573 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.922553 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fvttb" Jul 27 21:54:39.928521 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.928506 2559 manager.go:354] Recovery completed Jul 27 21:54:39.931219 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.931201 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fvttb" Jul 27 21:54:39.932776 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.932763 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:39.935194 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935180 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:39.935256 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935206 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:39.935256 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935217 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:39.935776 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935765 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:39.935817 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935767 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 21:54:39.935851 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935823 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 21:54:39.935851 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.935840 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:54:39.936454 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.936441 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:39.936497 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.936467 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:39.936497 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.936481 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:39.937141 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.937130 2559 policy_none.go:49] "None policy: Start" Jul 27 21:54:39.937192 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.937145 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 21:54:39.937192 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.937173 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 21:54:39.937591 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.937530 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-139.ec2.internal.18c64431175f6221 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-139.ec2.internal,UID:ip-10-0-134-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-139.ec2.internal,},FirstTimestamp:2026-07-27 21:54:39.935193633 +0000 UTC m=+0.316265547,LastTimestamp:2026-07-27 21:54:39.935193633 +0000 UTC m=+0.316265547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-139.ec2.internal,}" Jul 27 21:54:39.938078 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.938067 2559 policy_none.go:47] "Start" Jul 27 21:54:39.975890 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.975731 2559 manager.go:337] "Starting Device Plugin manager" Jul 27 21:54:39.975991 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.975919 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 21:54:39.975991 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.975934 2559 server.go:87] "Starting device plugin registration server" Jul 27 21:54:39.976198 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.976187 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 21:54:39.976239 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.976199 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 21:54:39.976344 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.976327 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 21:54:39.976687 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.976476 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 21:54:39.976687 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:39.976491 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 21:54:39.976960 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.976942 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 21:54:39.977051 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:39.976992 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.076271 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.076230 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.076302 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077369 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077407 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077422 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077457 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077656 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077698 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.077720 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 21:54:40.078490 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.077752 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 21:54:40.079973 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.079951 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:54:40.084040 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.084024 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.084128 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.084045 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-139.ec2.internal\": node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.151070 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.151042 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.178842 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.178800 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal"] Jul 27 21:54:40.178972 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.178890 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:40.180763 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.180744 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:40.180841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.180784 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:40.180841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.180795 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:40.183246 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.183232 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:40.183381 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.183368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.183428 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.183396 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:40.184037 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184013 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:40.184142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184048 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:40.184142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184019 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:40.184142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184085 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:40.184142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184061 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:40.184142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.184105 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:40.186457 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.186444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.186520 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.186468 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:54:40.187165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.187132 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:54:40.187248 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.187174 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:54:40.187248 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.187196 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:54:40.209759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.209737 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-139.ec2.internal\" not found" node="ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.214092 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.214075 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-139.ec2.internal\" not found" node="ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.251561 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.251534 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.318288 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.318259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/103daea59f1aca008ae67124379b3014-config\") pod \"kube-apiserver-proxy-ip-10-0-134-139.ec2.internal\" (UID: \"103daea59f1aca008ae67124379b3014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.318408 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.318293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.318408 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.318318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.352520 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.352492 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.419172 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.419172 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.419172 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/103daea59f1aca008ae67124379b3014-config\") pod \"kube-apiserver-proxy-ip-10-0-134-139.ec2.internal\" (UID: \"103daea59f1aca008ae67124379b3014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.419354 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.419354 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4bb3ef631eca2b289c89f62b6eba03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal\" (UID: \"9f4bb3ef631eca2b289c89f62b6eba03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.419354 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.419200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/103daea59f1aca008ae67124379b3014-config\") pod \"kube-apiserver-proxy-ip-10-0-134-139.ec2.internal\" (UID: \"103daea59f1aca008ae67124379b3014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.453229 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.453196 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.512385 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.512361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.517409 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.517276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:40.553528 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.553500 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.654014 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.653963 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.754544 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.754469 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.854979 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.854930 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.865124 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.865093 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 21:54:40.865277 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.865263 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:54:40.935837 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.935798 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 21:49:39 +0000 UTC" deadline="2028-02-13 12:59:18.696554601 +0000 UTC" Jul 27 21:54:40.935837 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.935833 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13575h4m37.760725552s" Jul 27 21:54:40.956005 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:40.955972 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:40.976388 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:40.976361 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:54:41.056593 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:41.056522 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-139.ec2.internal\" not found" Jul 27 21:54:41.088457 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:41.088423 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4bb3ef631eca2b289c89f62b6eba03.slice/crio-db637a9744e5b34382b6c190d5c64df57067f07857458294382b861fa7af757d WatchSource:0}: Error finding container db637a9744e5b34382b6c190d5c64df57067f07857458294382b861fa7af757d: Status 404 returned error can't find the container with id db637a9744e5b34382b6c190d5c64df57067f07857458294382b861fa7af757d Jul 27 21:54:41.088914 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:41.088891 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103daea59f1aca008ae67124379b3014.slice/crio-640d68cc031c68e55306a48d86b0d811443d18a25eff5348e9d968a5cd871bc7 WatchSource:0}: Error finding container 640d68cc031c68e55306a48d86b0d811443d18a25eff5348e9d968a5cd871bc7: Status 404 returned error can't find the container with id 640d68cc031c68e55306a48d86b0d811443d18a25eff5348e9d968a5cd871bc7 Jul 27 21:54:41.092547 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.092532 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:54:41.142751 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.142725 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:54:41.217676 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.217646 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" Jul 27 21:54:41.230588 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.230565 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:54:41.232286 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.232272 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" Jul 27 21:54:41.242547 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.242522 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:54:41.398974 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.398939 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:54:41.896741 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.896714 2559 apiserver.go:52] "Watching apiserver" Jul 27 21:54:41.906712 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.906258 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 21:54:41.907810 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.907781 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp","openshift-cluster-node-tuning-operator/tuned-vvtpm","openshift-multus/multus-kr2lz","openshift-network-diagnostics/network-check-target-st8c7","openshift-network-operator/iptables-alerter-kkvqn","openshift-ovn-kubernetes/ovnkube-node-8q96t","kube-system/global-pull-secret-syncer-grl2x","openshift-dns/node-resolver-dzr5j","openshift-image-registry/node-ca-h5nx9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-7jb6g","openshift-multus/network-metrics-daemon-xzk78","kube-system/konnectivity-agent-bdz5l"] Jul 27 21:54:41.910717 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.910693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.913693 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.913406 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:41.913693 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:41.913498 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.914367 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.914477 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.914367 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.914851 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.915080 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.915352 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.915670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.918182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.916601 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jrdj5\"" Jul 27 21:54:41.918585 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.918404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.919030 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.918956 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 21:54:41.919569 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.919468 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.919569 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.919499 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.919569 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.919521 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2b7dr\"" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.920653 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.920680 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.920868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.920932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:41.920991 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.921224 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.921539 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.921252 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6xcrd\"" Jul 27 21:54:41.923480 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.922745 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:41.925701 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.925678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:41.927955 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.927932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-conf-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.927967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-netns\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.927994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7666fe1e-561d-431f-b335-ff0425b03b60-ovn-node-metrics-cert\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928021 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-conf\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-host\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-kubernetes\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-kubelet\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-bin\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-script-lib\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-sys\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-lib-modules\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-tuned\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5zl\" (UniqueName: \"kubernetes.io/projected/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-kube-api-access-lm5zl\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w76g\" (UniqueName: \"kubernetes.io/projected/1f243f54-1d60-4ff0-919f-86406da12c9d-kube-api-access-6w76g\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928297 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-systemd-units\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928344 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-etc-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysconfig\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-daemon-config\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-var-lib-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928427 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-netns\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928442 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-hostroot\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-systemd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-ovn\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-socket-dir-parent\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-multus-certs\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-netd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2x8z\" (UniqueName: \"kubernetes.io/projected/7666fe1e-561d-431f-b335-ff0425b03b60-kube-api-access-m2x8z\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-modprobe-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-var-lib-kubelet\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-cnibin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.928704 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-kubelet\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-bin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-cni-binary-copy\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-multus\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-node-log\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-systemd\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-run\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-etc-kubernetes\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhs5\" (UniqueName: \"kubernetes.io/projected/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-kube-api-access-5mhs5\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.928991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-tmp\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-os-release\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-k8s-cni-cncf-io\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-env-overrides\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929101 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-system-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-slash\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-log-socket\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-config\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:41.929921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929438 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.930373 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.929955 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wvqfm\"" Jul 27 21:54:41.931016 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.930993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:41.932206 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.932189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 21:54:41.932475 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.932457 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.932875 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.932855 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wdn2g\"" Jul 27 21:54:41.933304 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.933285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:41.935562 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.935544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:41.935644 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:41.935609 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:41.936402 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.936384 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.938185 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.938151 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:41.940526 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.940506 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.943962 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.943891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.944544 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.944605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.944791 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wktfj\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.944816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.944944 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.945047 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.945070 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 21:54:41.945319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.945125 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9pvkt\"" Jul 27 21:54:41.945757 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.945474 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x5s22\"" Jul 27 21:54:41.946192 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.946006 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 21:54:41.949256 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.948766 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 21:54:41.949256 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.949000 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 21:54:41.950181 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.949964 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:54:41.953550 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.953337 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x57hn\"" Jul 27 21:54:41.964874 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.964853 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:54:41.994220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:41.994193 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9l7cx" Jul 27 21:54:42.018182 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.018155 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 21:54:42.018493 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.018468 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9l7cx" Jul 27 21:54:42.029928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-multus-certs\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.029928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-netd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.029928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgc6\" (UniqueName: \"kubernetes.io/projected/5343a980-96ea-4633-9a06-5511934c207f-kube-api-access-ldgc6\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.029928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-multus-certs\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.029928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3df729f-a496-4eab-bde4-60eb6e6cffe7-agent-certs\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029941 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2x8z\" (UniqueName: \"kubernetes.io/projected/7666fe1e-561d-431f-b335-ff0425b03b60-kube-api-access-m2x8z\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-host\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-modprobe-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-var-lib-kubelet\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-cnibin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-kubelet\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030111 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-sys-fs\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030177 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-dbus\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-bin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3df729f-a496-4eab-bde4-60eb6e6cffe7-konnectivity-ca\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzqr\" (UniqueName: \"kubernetes.io/projected/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kube-api-access-6jzqr\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-cni-binary-copy\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-multus\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.030339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8tq\" (UniqueName: \"kubernetes.io/projected/b0ecbb3b-3093-49be-8dd6-2a2195103369-kube-api-access-lx8tq\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5343a980-96ea-4633-9a06-5511934c207f-hosts-file\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-node-log\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-systemd\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030486 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0ecbb3b-3093-49be-8dd6-2a2195103369-iptables-alerter-script\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030547 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-serviceca\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-run\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-etc-kubernetes\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhs5\" (UniqueName: \"kubernetes.io/projected/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-kube-api-access-5mhs5\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-tmp\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-os-release\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031157 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cnibin\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-k8s-cni-cncf-io\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-env-overrides\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-system-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-slash\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.030991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-log-socket\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-config\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031042 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-conf-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-netns\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7666fe1e-561d-431f-b335-ff0425b03b60-ovn-node-metrics-cert\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031149 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031174 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5343a980-96ea-4633-9a06-5511934c207f-tmp-dir\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-conf\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-host\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-registration-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.031997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-kubelet-config\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-kubernetes\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-kubelet\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-bin\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-script-lib\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031442 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-sys\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-lib-modules\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-tuned\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5zl\" (UniqueName: \"kubernetes.io/projected/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-kube-api-access-lm5zl\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w76g\" (UniqueName: \"kubernetes.io/projected/1f243f54-1d60-4ff0-919f-86406da12c9d-kube-api-access-6w76g\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-systemd-units\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-etc-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-system-cni-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysconfig\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-daemon-config\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-var-lib-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.032758 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-os-release\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0ecbb3b-3093-49be-8dd6-2a2195103369-host-slash\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwth2\" (UniqueName: \"kubernetes.io/projected/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-kube-api-access-pwth2\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031903 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-socket-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-netns\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-hostroot\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.031984 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-systemd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-ovn\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crv8q\" (UniqueName: \"kubernetes.io/projected/41bb9b0c-eb9b-45df-8601-3ef7e1563049-kube-api-access-crv8q\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-device-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-socket-dir-parent\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-socket-dir-parent\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.029961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-netd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-modprobe-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-var-lib-kubelet\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.033600 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.032866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-cnibin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.033125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-kubelet\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.033226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-bin\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.033855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-cni-binary-copy\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.033938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-cni-multus\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034006 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-node-log\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-systemd\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034320 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-d\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.034412 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-run\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.034935 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.034935 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.034548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-etc-kubernetes\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.035116 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-os-release\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.035116 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-sys\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.035305 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-ovn-kubernetes\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035305 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-lib-modules\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.035305 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-k8s-cni-cncf-io\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.035680 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-env-overrides\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035789 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-system-cni-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.035789 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-systemd-units\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035789 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-etc-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035975 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-slash\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035975 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-log-socket\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.035975 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.035874 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysconfig\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.036301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-daemon-config\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.036388 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-config\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.036388 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-var-lib-openvswitch\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.036493 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-host\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.036493 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.036420 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:42.036493 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-multus-conf-dir\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.036493 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-run-netns\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.036493 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.036484 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:54:42.536463914 +0000 UTC m=+2.917535837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:42.036752 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-systemd\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.036860 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036810 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-run-netns\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.036860 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.036854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-hostroot\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037012 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-sysctl-conf\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037084 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-run-ovn\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7666fe1e-561d-431f-b335-ff0425b03b60-ovnkube-script-lib\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-kubernetes\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f243f54-1d60-4ff0-919f-86406da12c9d-host-var-lib-kubelet\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7666fe1e-561d-431f-b335-ff0425b03b60-host-cni-bin\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.037896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.037406 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 21:54:42.042434 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.042348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-etc-tuned\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.042434 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.042375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-tmp\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.043075 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.043004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7666fe1e-561d-431f-b335-ff0425b03b60-ovn-node-metrics-cert\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.045104 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.044459 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:42.045104 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.044481 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:42.045104 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.044495 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:42.045104 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.044556 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:54:42.544536791 +0000 UTC m=+2.925608708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:42.049053 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.048979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w76g\" (UniqueName: \"kubernetes.io/projected/1f243f54-1d60-4ff0-919f-86406da12c9d-kube-api-access-6w76g\") pod \"multus-kr2lz\" (UID: \"1f243f54-1d60-4ff0-919f-86406da12c9d\") " pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.050126 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.050031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2x8z\" (UniqueName: \"kubernetes.io/projected/7666fe1e-561d-431f-b335-ff0425b03b60-kube-api-access-m2x8z\") pod \"ovnkube-node-8q96t\" (UID: \"7666fe1e-561d-431f-b335-ff0425b03b60\") " pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.050126 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.050083 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5zl\" (UniqueName: \"kubernetes.io/projected/ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6-kube-api-access-lm5zl\") pod \"tuned-vvtpm\" (UID: \"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6\") " pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.054591 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.054565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhs5\" (UniqueName: \"kubernetes.io/projected/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-kube-api-access-5mhs5\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:42.084881 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.084782 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" event={"ID":"103daea59f1aca008ae67124379b3014","Type":"ContainerStarted","Data":"640d68cc031c68e55306a48d86b0d811443d18a25eff5348e9d968a5cd871bc7"} Jul 27 21:54:42.086294 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.086264 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" event={"ID":"9f4bb3ef631eca2b289c89f62b6eba03","Type":"ContainerStarted","Data":"db637a9744e5b34382b6c190d5c64df57067f07857458294382b861fa7af757d"} Jul 27 21:54:42.132473 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cnibin\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5343a980-96ea-4633-9a06-5511934c207f-tmp-dir\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-registration-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-kubelet-config\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-system-cni-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-os-release\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0ecbb3b-3093-49be-8dd6-2a2195103369-host-slash\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwth2\" (UniqueName: \"kubernetes.io/projected/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-kube-api-access-pwth2\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-socket-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132675 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crv8q\" (UniqueName: \"kubernetes.io/projected/41bb9b0c-eb9b-45df-8601-3ef7e1563049-kube-api-access-crv8q\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-device-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgc6\" (UniqueName: \"kubernetes.io/projected/5343a980-96ea-4633-9a06-5511934c207f-kube-api-access-ldgc6\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3df729f-a496-4eab-bde4-60eb6e6cffe7-agent-certs\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-host\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-sys-fs\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-dbus\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3df729f-a496-4eab-bde4-60eb6e6cffe7-konnectivity-ca\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzqr\" (UniqueName: \"kubernetes.io/projected/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kube-api-access-6jzqr\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8tq\" (UniqueName: \"kubernetes.io/projected/b0ecbb3b-3093-49be-8dd6-2a2195103369-kube-api-access-lx8tq\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5343a980-96ea-4633-9a06-5511934c207f-hosts-file\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.132947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0ecbb3b-3093-49be-8dd6-2a2195103369-iptables-alerter-script\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.132997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-serviceca\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-device-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133299 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-system-cni-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133334 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-dbus\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0ecbb3b-3093-49be-8dd6-2a2195103369-host-slash\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133382 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-os-release\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.133451 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-serviceca\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.133586 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.133520 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:54:42.633501422 +0000 UTC m=+3.014573328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-socket-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.133950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-sys-fs\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a851d07-33d7-4bcc-99e2-eb277ac875d4-kubelet-config\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-registration-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.134109 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3df729f-a496-4eab-bde4-60eb6e6cffe7-konnectivity-ca\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cnibin\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-host\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5343a980-96ea-4633-9a06-5511934c207f-hosts-file\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41bb9b0c-eb9b-45df-8601-3ef7e1563049-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.134449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5343a980-96ea-4633-9a06-5511934c207f-tmp-dir\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.134894 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.134790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0ecbb3b-3093-49be-8dd6-2a2195103369-iptables-alerter-script\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.135122 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.135098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.135284 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.135261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41bb9b0c-eb9b-45df-8601-3ef7e1563049-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.137191 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.137142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3df729f-a496-4eab-bde4-60eb6e6cffe7-agent-certs\") pod \"konnectivity-agent-bdz5l\" (UID: \"c3df729f-a496-4eab-bde4-60eb6e6cffe7\") " pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.143910 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.143886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgc6\" (UniqueName: \"kubernetes.io/projected/5343a980-96ea-4633-9a06-5511934c207f-kube-api-access-ldgc6\") pod \"node-resolver-dzr5j\" (UID: \"5343a980-96ea-4633-9a06-5511934c207f\") " pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.144118 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.144104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8tq\" (UniqueName: \"kubernetes.io/projected/b0ecbb3b-3093-49be-8dd6-2a2195103369-kube-api-access-lx8tq\") pod \"iptables-alerter-kkvqn\" (UID: \"b0ecbb3b-3093-49be-8dd6-2a2195103369\") " pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.151844 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.151774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crv8q\" (UniqueName: \"kubernetes.io/projected/41bb9b0c-eb9b-45df-8601-3ef7e1563049-kube-api-access-crv8q\") pod \"multus-additional-cni-plugins-7jb6g\" (UID: \"41bb9b0c-eb9b-45df-8601-3ef7e1563049\") " pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.152208 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.152170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwth2\" (UniqueName: \"kubernetes.io/projected/8f3b1d48-c616-4c79-8405-fe3c3f5cb03f-kube-api-access-pwth2\") pod \"node-ca-h5nx9\" (UID: \"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f\") " pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.152611 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.152591 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzqr\" (UniqueName: \"kubernetes.io/projected/bcc5e931-2803-4e7a-b552-3e1bade69d1c-kube-api-access-6jzqr\") pod \"aws-ebs-csi-driver-node-9k8bp\" (UID: \"bcc5e931-2803-4e7a-b552-3e1bade69d1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.236044 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.236003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:54:42.252589 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.252552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" Jul 27 21:54:42.263421 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.263215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kr2lz" Jul 27 21:54:42.273358 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.273333 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kkvqn" Jul 27 21:54:42.283274 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.283241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzr5j" Jul 27 21:54:42.292215 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.292185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:54:42.301290 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.301250 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" Jul 27 21:54:42.310228 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.310203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5nx9" Jul 27 21:54:42.318339 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.318316 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" Jul 27 21:54:42.346973 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.346934 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc5e931_2803_4e7a_b552_3e1bade69d1c.slice/crio-2bf134df44fdd23813bb6b493d29af14a89208a19b653206414081e5ca945d8f WatchSource:0}: Error finding container 2bf134df44fdd23813bb6b493d29af14a89208a19b653206414081e5ca945d8f: Status 404 returned error can't find the container with id 2bf134df44fdd23813bb6b493d29af14a89208a19b653206414081e5ca945d8f Jul 27 21:54:42.347401 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.347374 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f243f54_1d60_4ff0_919f_86406da12c9d.slice/crio-87115ead375751f37c6e56ed08d58b567c1d462b18cdf322c5ae98a8786865a5 WatchSource:0}: Error finding container 87115ead375751f37c6e56ed08d58b567c1d462b18cdf322c5ae98a8786865a5: Status 404 returned error can't find the container with id 87115ead375751f37c6e56ed08d58b567c1d462b18cdf322c5ae98a8786865a5 Jul 27 21:54:42.349813 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.349792 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7666fe1e_561d_431f_b335_ff0425b03b60.slice/crio-57a5f26feb8c3081639f06c84467dca8cd463c0233d859c42b5a84ffd063ea35 WatchSource:0}: Error finding container 57a5f26feb8c3081639f06c84467dca8cd463c0233d859c42b5a84ffd063ea35: Status 404 returned error can't find the container with id 57a5f26feb8c3081639f06c84467dca8cd463c0233d859c42b5a84ffd063ea35 Jul 27 21:54:42.371319 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.371287 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ecbb3b_3093_49be_8dd6_2a2195103369.slice/crio-f828032d936060bdfcba6357cef2da99297e54d4e4b0cc72ded7cc1db4d12aa9 WatchSource:0}: Error finding container f828032d936060bdfcba6357cef2da99297e54d4e4b0cc72ded7cc1db4d12aa9: Status 404 returned error can't find the container with id f828032d936060bdfcba6357cef2da99297e54d4e4b0cc72ded7cc1db4d12aa9 Jul 27 21:54:42.636089 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.636049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.636110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:42.636175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636232 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636252 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636255 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636266 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:42.636301 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636266 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:42.636586 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636318 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:54:43.63630174 +0000 UTC m=+4.017373649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:42.636586 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636336 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:54:43.636327327 +0000 UTC m=+4.017399235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:42.636586 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:42.636353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:54:43.636344476 +0000 UTC m=+4.017416380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:42.697815 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.697784 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5343a980_96ea_4633_9a06_5511934c207f.slice/crio-689429cfba773c4db6645335fbf10e63bb1caf46a421ae7244db7aac84741ed7 WatchSource:0}: Error finding container 689429cfba773c4db6645335fbf10e63bb1caf46a421ae7244db7aac84741ed7: Status 404 returned error can't find the container with id 689429cfba773c4db6645335fbf10e63bb1caf46a421ae7244db7aac84741ed7 Jul 27 21:54:42.698352 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.698330 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3df729f_a496_4eab_bde4_60eb6e6cffe7.slice/crio-6fb80213772cfc4f80ccd1cce0e04cafa8df0b5b0afe7da598449c0443d1cb37 WatchSource:0}: Error finding container 6fb80213772cfc4f80ccd1cce0e04cafa8df0b5b0afe7da598449c0443d1cb37: Status 404 returned error can't find the container with id 6fb80213772cfc4f80ccd1cce0e04cafa8df0b5b0afe7da598449c0443d1cb37 Jul 27 21:54:42.699755 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.699629 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b1d48_c616_4c79_8405_fe3c3f5cb03f.slice/crio-3f85e087afda3bcae80110f3b267848f89707e1ba091fc623d61d8fda48f7fda WatchSource:0}: Error finding container 3f85e087afda3bcae80110f3b267848f89707e1ba091fc623d61d8fda48f7fda: Status 404 returned error can't find the container with id 3f85e087afda3bcae80110f3b267848f89707e1ba091fc623d61d8fda48f7fda Jul 27 21:54:42.700285 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:54:42.700215 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac77b6cf_f7fa_4d85_ae4b_19daa3cd6bb6.slice/crio-d004df7d14933959d0ef0ef772e5b2b6aaed21cdf0fe34b3159c3252e7e7e75e WatchSource:0}: Error finding container d004df7d14933959d0ef0ef772e5b2b6aaed21cdf0fe34b3159c3252e7e7e75e: Status 404 returned error can't find the container with id d004df7d14933959d0ef0ef772e5b2b6aaed21cdf0fe34b3159c3252e7e7e75e Jul 27 21:54:43.022342 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.022245 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:49:42 +0000 UTC" deadline="2027-12-22 13:09:27.771073908 +0000 UTC" Jul 27 21:54:43.022342 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.022287 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12303h14m44.7487908s" Jul 27 21:54:43.078642 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.078614 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:43.078778 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.078751 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:43.091348 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.091311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"57a5f26feb8c3081639f06c84467dca8cd463c0233d859c42b5a84ffd063ea35"} Jul 27 21:54:43.094645 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.094622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5nx9" event={"ID":"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f","Type":"ContainerStarted","Data":"3f85e087afda3bcae80110f3b267848f89707e1ba091fc623d61d8fda48f7fda"} Jul 27 21:54:43.098098 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.098068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" event={"ID":"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6","Type":"ContainerStarted","Data":"d004df7d14933959d0ef0ef772e5b2b6aaed21cdf0fe34b3159c3252e7e7e75e"} Jul 27 21:54:43.099777 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.099749 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kkvqn" event={"ID":"b0ecbb3b-3093-49be-8dd6-2a2195103369","Type":"ContainerStarted","Data":"f828032d936060bdfcba6357cef2da99297e54d4e4b0cc72ded7cc1db4d12aa9"} Jul 27 21:54:43.102499 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.102471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kr2lz" event={"ID":"1f243f54-1d60-4ff0-919f-86406da12c9d","Type":"ContainerStarted","Data":"87115ead375751f37c6e56ed08d58b567c1d462b18cdf322c5ae98a8786865a5"} Jul 27 21:54:43.103782 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.103757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" event={"ID":"bcc5e931-2803-4e7a-b552-3e1bade69d1c","Type":"ContainerStarted","Data":"2bf134df44fdd23813bb6b493d29af14a89208a19b653206414081e5ca945d8f"} Jul 27 21:54:43.105053 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.105032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerStarted","Data":"edb34eb7c012cce8b9afc08dcfcee9112f34571da37b4f1b808fd00d9d743d71"} Jul 27 21:54:43.106292 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.106269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdz5l" event={"ID":"c3df729f-a496-4eab-bde4-60eb6e6cffe7","Type":"ContainerStarted","Data":"6fb80213772cfc4f80ccd1cce0e04cafa8df0b5b0afe7da598449c0443d1cb37"} Jul 27 21:54:43.107479 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.107457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzr5j" event={"ID":"5343a980-96ea-4633-9a06-5511934c207f","Type":"ContainerStarted","Data":"689429cfba773c4db6645335fbf10e63bb1caf46a421ae7244db7aac84741ed7"} Jul 27 21:54:43.644359 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.644082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:43.644551 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644268 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:43.644551 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.644429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:43.644551 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:43.644468 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:43.644551 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644511 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:54:45.644477298 +0000 UTC m=+6.025549213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644577 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644600 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644613 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644661 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:54:45.644646283 +0000 UTC m=+6.025718202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644730 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:43.644759 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:43.644762 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:54:45.644751472 +0000 UTC m=+6.025823389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:44.022838 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.022681 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:49:42 +0000 UTC" deadline="2028-04-08 10:27:53.950915008 +0000 UTC" Jul 27 21:54:44.022838 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.022747 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14892h33m9.928172822s" Jul 27 21:54:44.082254 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.081497 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:44.082254 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:44.081617 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:44.082254 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.082023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:44.082254 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:44.082113 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:44.123183 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.123125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" event={"ID":"103daea59f1aca008ae67124379b3014","Type":"ContainerStarted","Data":"cc0ed47dec62cace56fcfff85cd91222cb42d4d4eea2f77840c3560721e9c425"} Jul 27 21:54:44.135144 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.135107 2559 generic.go:356] "Generic (PLEG): container finished" podID="9f4bb3ef631eca2b289c89f62b6eba03" containerID="0b03c5dd36bf28d1c5d518155f46ab49a8099101637cb302d783a05a4d35c8aa" exitCode=0 Jul 27 21:54:44.135338 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.135186 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" event={"ID":"9f4bb3ef631eca2b289c89f62b6eba03","Type":"ContainerDied","Data":"0b03c5dd36bf28d1c5d518155f46ab49a8099101637cb302d783a05a4d35c8aa"} Jul 27 21:54:44.158012 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:44.157955 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-139.ec2.internal" podStartSLOduration=3.1579330150000002 podStartE2EDuration="3.157933015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:44.138930816 +0000 UTC m=+4.520002743" watchObservedRunningTime="2026-07-27 21:54:44.157933015 +0000 UTC m=+4.539004940" Jul 27 21:54:45.078286 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:45.078251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:45.078743 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.078408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:45.154440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:45.154403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" event={"ID":"9f4bb3ef631eca2b289c89f62b6eba03","Type":"ContainerStarted","Data":"c76e33351ecd0473db44c440b3e4f16244e94a805abc4d2a23101c97141fa5ca"} Jul 27 21:54:45.663981 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:45.663945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:45.664197 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:45.664002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:45.664197 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:45.664046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:45.664197 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664185 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:45.664371 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664250 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:54:49.66423206 +0000 UTC m=+10.045303967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:45.664371 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664313 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:45.664371 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:54:49.664335483 +0000 UTC m=+10.045407390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:45.664524 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664420 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:45.664524 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664435 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:45.664524 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664448 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:45.664524 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:45.664479 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:54:49.664469307 +0000 UTC m=+10.045541210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:46.080809 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:46.080120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:46.080809 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:46.080273 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:46.080809 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:46.080586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:46.080809 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:46.080685 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:47.078586 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:47.078537 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:47.078772 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:47.078685 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:48.080604 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:48.080570 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:48.081085 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:48.080691 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:48.081221 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:48.081106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:48.081291 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:48.081233 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:49.079342 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:49.078784 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:49.079342 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.078928 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:49.701195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:49.701256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:49.701302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701346 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701408 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701420 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:54:57.701401091 +0000 UTC m=+18.082473009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701423 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701442 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701453 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701463 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:54:57.701445271 +0000 UTC m=+18.082517176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:49.701495 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:49.701498 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:54:57.701475776 +0000 UTC m=+18.082547692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:50.080382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:50.080303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:50.080544 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:50.080417 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:50.080757 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:50.080740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:50.080841 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:50.080822 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:51.078440 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:51.078344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:51.078791 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:51.078491 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:52.078239 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:52.078199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:52.078239 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:52.078241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:52.078481 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:52.078342 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:52.078833 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:52.078479 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:53.078899 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:53.078864 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:53.079383 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:53.078981 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:54.078165 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:54.078071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:54.078330 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:54.078223 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:54.078330 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:54.078271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:54.078446 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:54.078375 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:55.078785 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:55.078752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:55.079219 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:55.078873 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:56.078225 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:56.078183 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:56.078225 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:56.078225 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:56.078434 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:56.078320 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:56.078434 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:56.078398 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:57.078398 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:57.078359 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:57.078843 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.078480 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:54:57.757801 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:57.757756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:57.757840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:57.757891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.757930 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.757957 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.757971 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:57.757997 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.757996 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:57.758308 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.757997 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:57.758308 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.758028 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:55:13.758012946 +0000 UTC m=+34.139084852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:54:57.758308 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.758050 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:13.75803527 +0000 UTC m=+34.139107187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:54:57.758308 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:57.758069 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:55:13.75805895 +0000 UTC m=+34.139130861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:54:58.078810 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:58.078726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:54:58.078810 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:58.078753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:54:58.079313 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:58.078868 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:54:58.079313 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:58.078989 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:54:59.078459 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:54:59.078418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:54:59.078631 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:54:59.078550 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:00.079654 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:00.079618 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:00.080036 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:00.079726 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:00.080036 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:00.079776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:00.080036 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:00.079885 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:01.078946 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:01.078911 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:01.079108 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:01.079041 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:02.078429 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.078114 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:02.079266 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.078184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:02.079266 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:02.078517 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:02.079266 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:02.078587 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:02.193302 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.193270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5nx9" event={"ID":"8f3b1d48-c616-4c79-8405-fe3c3f5cb03f","Type":"ContainerStarted","Data":"ae554c725ce6199ac929ce11e4067e5986bca8331c0b1fab651765e55cc2b6e3"} Jul 27 21:55:02.194724 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.194695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" event={"ID":"ac77b6cf-f7fa-4d85-ae4b-19daa3cd6bb6","Type":"ContainerStarted","Data":"a8ac4bceca7a37cacf58a63fd01035635b58295fdd75ac3695c6496bba714742"} Jul 27 21:55:02.196089 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.196069 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kr2lz" event={"ID":"1f243f54-1d60-4ff0-919f-86406da12c9d","Type":"ContainerStarted","Data":"4a9e4699c1e944efad9f924a7e27b6af6316422037862ac6ce067238996adf14"} Jul 27 21:55:02.197437 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.197418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" event={"ID":"bcc5e931-2803-4e7a-b552-3e1bade69d1c","Type":"ContainerStarted","Data":"48f1e3edc843c93ec4fae0e872f8ebee7295630e6f076c010f2f4b5aa5c5b676"} Jul 27 21:55:02.198969 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.198947 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="5ac52cdccee252f4c8c3641e74502ab75d125daa9f965caaf8f1db698b3e5dca" exitCode=0 Jul 27 21:55:02.199058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.198988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"5ac52cdccee252f4c8c3641e74502ab75d125daa9f965caaf8f1db698b3e5dca"} Jul 27 21:55:02.200607 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.200574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdz5l" event={"ID":"c3df729f-a496-4eab-bde4-60eb6e6cffe7","Type":"ContainerStarted","Data":"cfb3b768abd6f10ed88d10ad1695920d12005f69bc558d03a0a8815e431197b8"} Jul 27 21:55:02.201822 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.201798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzr5j" event={"ID":"5343a980-96ea-4633-9a06-5511934c207f","Type":"ContainerStarted","Data":"72b1a2703d78c4f0070ccf5a09b0cab487ac5c202be0fe825cebcc74fc7217a2"} Jul 27 21:55:02.203816 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.203799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"0dc0f103ce551aa151ae0aaa3a5dc5b79edfce9712ecf9eeadf2a65cf725a9cd"} Jul 27 21:55:02.203880 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.203851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"6b05485e78677109f2948d602ba95a672ec0d01f9cc49e3b834c221241193056"} Jul 27 21:55:02.203928 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.203881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"459ce404234018041e9af2d17fca3b8162433ff05456d907c6e7f3da489cdeaf"} Jul 27 21:55:02.211582 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.211542 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h5nx9" podStartSLOduration=3.151194989 podStartE2EDuration="22.21153064s" totalImagesPullingTime="19.060335651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:02.211336107 +0000 UTC m=+22.592408029" watchObservedRunningTime="2026-07-27 21:55:02.21153064 +0000 UTC m=+22.592602563" Jul 27 21:55:02.211703 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.211629 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-139.ec2.internal" podStartSLOduration=21.211619819 podStartE2EDuration="21.211619819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:45.170519606 +0000 UTC m=+5.551591530" watchObservedRunningTime="2026-07-27 21:55:02.211619819 +0000 UTC m=+22.592691745" Jul 27 21:55:02.228348 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.228303 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dzr5j" podStartSLOduration=3.203488138 podStartE2EDuration="22.228286639s" totalImagesPullingTime="19.024798501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:02.226288102 +0000 UTC m=+22.607360027" watchObservedRunningTime="2026-07-27 21:55:02.228286639 +0000 UTC m=+22.609358563" Jul 27 21:55:02.241172 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.241094 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdz5l" podStartSLOduration=3.178294727 podStartE2EDuration="22.241080029s" totalImagesPullingTime="19.062785302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:02.240758167 +0000 UTC m=+22.621830091" watchObservedRunningTime="2026-07-27 21:55:02.241080029 +0000 UTC m=+22.622151952" Jul 27 21:55:02.263017 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.262978 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kr2lz" podStartSLOduration=2.850313008 podStartE2EDuration="22.26295885s" totalImagesPullingTime="19.412645842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:02.262537878 +0000 UTC m=+22.643609804" watchObservedRunningTime="2026-07-27 21:55:02.26295885 +0000 UTC m=+22.644030774" Jul 27 21:55:02.311095 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.311007 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvtpm" podStartSLOduration=3.250862607 podStartE2EDuration="22.310993635s" totalImagesPullingTime="19.060131028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:02.310578862 +0000 UTC m=+22.691650786" watchObservedRunningTime="2026-07-27 21:55:02.310993635 +0000 UTC m=+22.692065559" Jul 27 21:55:02.421316 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.421282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:02.497094 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.497060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:02.935842 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.935808 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:55:02.936485 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:02.936466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:55:03.078708 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.078685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:03.079060 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:03.078799 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:03.106947 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.106917 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 21:55:03.210446 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.210418 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"7dc6f691079b36877ddd00c8ae366ad07d414c6c69eb15033207df8dd1b999b9"} Jul 27 21:55:03.210579 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.210507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"3741c0441c3c3cfffe3b17ffb8396748b75ce7729b74d8c024183055366ec88e"} Jul 27 21:55:03.210579 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.210560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"0caead929f47640c65938164099db319088c0ab1547bd1a1f25a9677cfea19ba"} Jul 27 21:55:03.215429 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.215402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" event={"ID":"bcc5e931-2803-4e7a-b552-3e1bade69d1c","Type":"ContainerStarted","Data":"775e40d91162cab0f041301714752a9b8f4183d46b25d8de173d58d49ae3ffbe"} Jul 27 21:55:03.215931 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.215831 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:55:03.216573 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.216557 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdz5l" Jul 27 21:55:03.369467 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.369440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:03.991372 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.991274 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:55:03.106940428Z","UUID":"8eb36232-df6c-4363-9e13-108a72a5b5c9","Handler":null,"Name":"","Endpoint":""} Jul 27 21:55:03.992987 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.992963 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 21:55:03.993133 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:03.993004 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 21:55:04.083813 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.083784 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:04.084252 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.083781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:04.084252 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:04.083952 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:04.084252 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:04.084096 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:04.219718 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.219678 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kkvqn" event={"ID":"b0ecbb3b-3093-49be-8dd6-2a2195103369","Type":"ContainerStarted","Data":"bdb08ebeae79d8538150ed479ca9dd788b71dee0dcdcb67709c405bb71e9645d"} Jul 27 21:55:04.222294 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.222264 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" event={"ID":"bcc5e931-2803-4e7a-b552-3e1bade69d1c","Type":"ContainerStarted","Data":"0223fdb7e0250eb39cffa3f7bac320e7767ca2bf4b15a4e586c092127e9c1903"} Jul 27 21:55:04.234471 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.234415 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kkvqn" podStartSLOduration=4.846455827 podStartE2EDuration="24.234397797s" totalImagesPullingTime="19.38794197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:04.234300336 +0000 UTC m=+24.615372262" watchObservedRunningTime="2026-07-27 21:55:04.234397797 +0000 UTC m=+24.615469722" Jul 27 21:55:04.251458 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.251357 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k8bp" podStartSLOduration=7.327179752 podStartE2EDuration="24.251337114s" totalImagesPullingTime="16.924157362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:04.250498408 +0000 UTC m=+24.631570332" watchObservedRunningTime="2026-07-27 21:55:04.251337114 +0000 UTC m=+24.632409038" Jul 27 21:55:04.356220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.356189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:04.456390 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:04.456360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:05.078760 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:05.078719 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:05.078923 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:05.078867 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:05.228364 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:05.228330 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"5ce3b3480ac2215c7111be337ecb2e9ffd373de23924032b124e6d7d0959d43f"} Jul 27 21:55:06.078394 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:06.078208 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:06.078549 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:06.078496 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:06.078549 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:06.078257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:06.078670 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:06.078589 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:07.078531 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.078503 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:07.079230 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:07.078638 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:07.234471 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.234225 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="10e809221716c67273c04f92b0bdbcbbbb7cb999e53aa1d26ab4ff619bfd83be" exitCode=0 Jul 27 21:55:07.234471 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.234303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"10e809221716c67273c04f92b0bdbcbbbb7cb999e53aa1d26ab4ff619bfd83be"} Jul 27 21:55:07.240117 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.240093 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" event={"ID":"7666fe1e-561d-431f-b335-ff0425b03b60","Type":"ContainerStarted","Data":"cc9a5f05a9f6d43ec2b88bace2633b5968bb052114f790cc9ad9ef9199469c12"} Jul 27 21:55:07.240307 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.240295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:07.240364 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.240355 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:07.256241 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.256209 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:07.282221 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:07.282176 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" podStartSLOduration=7.797813763 podStartE2EDuration="27.28214446s" totalImagesPullingTime="19.484330697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:07.281987939 +0000 UTC m=+27.663059863" watchObservedRunningTime="2026-07-27 21:55:07.28214446 +0000 UTC m=+27.663216384" Jul 27 21:55:08.078433 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.078409 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:08.078564 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.078411 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:08.078564 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:08.078540 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:08.078938 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:08.078639 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:08.245031 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.244952 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="f05fee714ca227355f450e68b3f83aaf7e6bfbd5e70b01910aa00d9cc1d89f19" exitCode=0 Jul 27 21:55:08.245201 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.245038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"f05fee714ca227355f450e68b3f83aaf7e6bfbd5e70b01910aa00d9cc1d89f19"} Jul 27 21:55:08.245544 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.245514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:08.261549 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:08.261527 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:09.078247 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.078215 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:09.078402 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:09.078322 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:09.138007 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.137974 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-st8c7"] Jul 27 21:55:09.138510 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.138097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:09.138510 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:09.138208 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:09.141729 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.141698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-grl2x"] Jul 27 21:55:09.141833 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.141760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzk78"] Jul 27 21:55:09.141881 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.141837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:09.141953 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:09.141923 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:09.247999 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:09.247972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:09.248199 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:09.248177 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:10.253234 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:10.252947 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="5ba4affcff2400902f05ab9f152a1beb18396c7216a9b8e4e5fda6904c805dd6" exitCode=0 Jul 27 21:55:10.253620 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:10.253019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"5ba4affcff2400902f05ab9f152a1beb18396c7216a9b8e4e5fda6904c805dd6"} Jul 27 21:55:11.078713 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:11.078619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:11.078713 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:11.078672 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:11.078713 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:11.078678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:11.078987 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:11.078754 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:11.078987 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:11.078852 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:11.078987 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:11.078931 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:13.078692 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.078616 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:13.079309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.078615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:13.079309 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.078730 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-st8c7" podUID="1304f40d-41b1-46bd-9849-a756bb24911d" Jul 27 21:55:13.079309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.078615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:13.079309 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.078822 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzk78" podUID="ff07cd87-c25a-4f0c-a5e8-4b172891c63b" Jul 27 21:55:13.079309 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.078933 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-grl2x" podUID="8a851d07-33d7-4bcc-99e2-eb277ac875d4" Jul 27 21:55:13.662322 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.662287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:13.759792 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.759757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:13.790744 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.790708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.790770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.790804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790833 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790901 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790916 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret podName:8a851d07-33d7-4bcc-99e2-eb277ac875d4 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:45.790891817 +0000 UTC m=+66.171963742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret") pod "global-pull-secret-syncer-grl2x" (UID: "8a851d07-33d7-4bcc-99e2-eb277ac875d4") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790919 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:55:13.790940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790937 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:55:13.791326 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790949 2559 projected.go:196] Error preparing data for projected volume kube-api-access-d9c2h for pod openshift-network-diagnostics/network-check-target-st8c7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:55:13.791326 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790952 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:55:45.790935454 +0000 UTC m=+66.172007371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:55:13.791326 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:13.790994 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h podName:1304f40d-41b1-46bd-9849-a756bb24911d nodeName:}" failed. No retries permitted until 2026-07-27 21:55:45.790985839 +0000 UTC m=+66.172057741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9c2h" (UniqueName: "kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h") pod "network-check-target-st8c7" (UID: "1304f40d-41b1-46bd-9849-a756bb24911d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:55:13.851045 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:13.851005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:14.475366 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.475340 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-139.ec2.internal" event="NodeReady" Jul 27 21:55:14.475908 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.475456 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 21:55:14.556439 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.556351 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hzf7h"] Jul 27 21:55:14.576629 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.576596 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b9kz5"] Jul 27 21:55:14.576764 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.576714 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.582414 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.582387 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jz2cl\"" Jul 27 21:55:14.582414 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.582413 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 21:55:14.583399 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.583376 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 21:55:14.590754 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.590731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzf7h"] Jul 27 21:55:14.590866 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.590819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.594065 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.594043 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 21:55:14.594200 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.594045 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 21:55:14.594200 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.594109 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c4sqs\"" Jul 27 21:55:14.594200 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.594109 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 21:55:14.603081 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.603058 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9kz5"] Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.696868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.696912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-cert\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.696953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f1f903-b9f0-468c-9a33-4294fa33d347-config-volume\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.696982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfm8l\" (UniqueName: \"kubernetes.io/projected/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-kube-api-access-kfm8l\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.697039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f1f903-b9f0-468c-9a33-4294fa33d347-tmp-dir\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.697142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.697064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrcf\" (UniqueName: \"kubernetes.io/projected/a6f1f903-b9f0-468c-9a33-4294fa33d347-kube-api-access-hhrcf\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.739797 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.739766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:14.797534 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.797670 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-cert\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.797670 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f1f903-b9f0-468c-9a33-4294fa33d347-config-volume\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.797670 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm8l\" (UniqueName: \"kubernetes.io/projected/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-kube-api-access-kfm8l\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.797670 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:14.797646 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:14.797878 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f1f903-b9f0-468c-9a33-4294fa33d347-tmp-dir\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.797878 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:14.797716 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:15.297698334 +0000 UTC m=+35.678770248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:14.797878 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.797750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrcf\" (UniqueName: \"kubernetes.io/projected/a6f1f903-b9f0-468c-9a33-4294fa33d347-kube-api-access-hhrcf\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.798083 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.798014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f1f903-b9f0-468c-9a33-4294fa33d347-tmp-dir\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.798326 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.798306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f1f903-b9f0-468c-9a33-4294fa33d347-config-volume\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.802420 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.802396 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-cert\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.809466 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.809411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrcf\" (UniqueName: \"kubernetes.io/projected/a6f1f903-b9f0-468c-9a33-4294fa33d347-kube-api-access-hhrcf\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:14.809875 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.809853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm8l\" (UniqueName: \"kubernetes.io/projected/e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6-kube-api-access-kfm8l\") pod \"ingress-canary-b9kz5\" (UID: \"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6\") " pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:14.841526 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.841484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:14.898769 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:14.898737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9kz5" Jul 27 21:55:15.078908 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.078681 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:15.079034 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.078698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:15.079034 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.078742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:15.082394 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.082375 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:55:15.083383 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.083225 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:55:15.083383 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.083248 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:55:15.083383 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.083259 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:55:15.083383 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.083274 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tm84h\"" Jul 27 21:55:15.083383 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.083302 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ffd2\"" Jul 27 21:55:15.102640 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.102614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9kz5"] Jul 27 21:55:15.107746 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:55:15.107723 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e538e7_d2a1_4e19_8fd3_f8a2d05fdbd6.slice/crio-91a63d2da63ec07f5901344b9ceb663cfb57ccccb702d48a052f4556d65f4d1b WatchSource:0}: Error finding container 91a63d2da63ec07f5901344b9ceb663cfb57ccccb702d48a052f4556d65f4d1b: Status 404 returned error can't find the container with id 91a63d2da63ec07f5901344b9ceb663cfb57ccccb702d48a052f4556d65f4d1b Jul 27 21:55:15.157630 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.157600 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:15.256769 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.256739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:15.266725 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.266692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9kz5" event={"ID":"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6","Type":"ContainerStarted","Data":"91a63d2da63ec07f5901344b9ceb663cfb57ccccb702d48a052f4556d65f4d1b"} Jul 27 21:55:15.272097 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.272073 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="473bd37a505a21173c2b97079328f85241d523ada4843f3c96d051b8999e66e6" exitCode=0 Jul 27 21:55:15.272219 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.272119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"473bd37a505a21173c2b97079328f85241d523ada4843f3c96d051b8999e66e6"} Jul 27 21:55:15.301533 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:15.301504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:15.301661 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:15.301645 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:15.301713 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:15.301703 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:16.301689306 +0000 UTC m=+36.682761211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:16.278628 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:16.278589 2559 generic.go:356] "Generic (PLEG): container finished" podID="41bb9b0c-eb9b-45df-8601-3ef7e1563049" containerID="627477d79ea88f1a306ecdce5fadd2e6fbd5f72bfc941b058be75b1119396dcc" exitCode=0 Jul 27 21:55:16.278628 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:16.278634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerDied","Data":"627477d79ea88f1a306ecdce5fadd2e6fbd5f72bfc941b058be75b1119396dcc"} Jul 27 21:55:16.310824 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:16.310794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:16.310955 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:16.310906 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:16.311009 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:16.310971 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:18.310954952 +0000 UTC m=+38.692026876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:17.285817 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:17.285636 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" event={"ID":"41bb9b0c-eb9b-45df-8601-3ef7e1563049","Type":"ContainerStarted","Data":"bb098a20086daefe0ac4578b6a057593358dcecca453c00c7f79227fc782edc6"} Jul 27 21:55:17.312340 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:17.312292 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7jb6g" podStartSLOduration=7.412729188 podStartE2EDuration="37.312265172s" totalImagesPullingTime="29.899535984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:17.312141802 +0000 UTC m=+37.693213725" watchObservedRunningTime="2026-07-27 21:55:17.312265172 +0000 UTC m=+37.693337100" Jul 27 21:55:17.438601 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:17.438556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:17.540377 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:17.540341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:18.290744 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:18.290295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9kz5" event={"ID":"e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6","Type":"ContainerStarted","Data":"c8bc56927d76d36f667a816e6601dc5a578ee15dfcecb9d229c3ff7223f04fcf"} Jul 27 21:55:18.307263 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:18.307215 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b9kz5" podStartSLOduration=1.913958593 podStartE2EDuration="4.30720009s" totalImagesPullingTime="2.393241497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:18.306952496 +0000 UTC m=+38.688024420" watchObservedRunningTime="2026-07-27 21:55:18.30720009 +0000 UTC m=+38.688272014" Jul 27 21:55:18.326509 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:18.326473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:18.326673 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:18.326623 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:18.326711 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:18.326697 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:22.326675057 +0000 UTC m=+42.707746973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:18.685636 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:18.685596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:19.719371 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:19.719290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:20.445467 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:20.445433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:22.357302 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:22.357264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:22.357714 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:22.357415 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:22.357714 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:22.357495 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:30.35747654 +0000 UTC m=+50.738548443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:25.816519 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:25.816476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:25.880960 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:25.880928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:29.787524 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:29.787475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:29.847744 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:29.847713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:29.913140 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:29.913105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:30.333170 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:30.333115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:30.395432 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:30.395404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:30.416562 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:30.416528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:30.416704 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:30.416675 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:30.416763 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:30.416751 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:46.416734436 +0000 UTC m=+66.797806338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:36.032483 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.032452 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr"] Jul 27 21:55:36.034199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.034182 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.036622 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.036595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 21:55:36.036731 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.036633 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 21:55:36.036731 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.036636 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 21:55:36.037566 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.037552 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xzw65\"" Jul 27 21:55:36.038647 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.038628 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 21:55:36.040905 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.040882 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr"] Jul 27 21:55:36.053899 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.053869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.054023 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.053953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f988c022-2428-446d-9ae1-71c26ae9bbfb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.054023 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.054005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8c9p\" (UniqueName: \"kubernetes.io/projected/f988c022-2428-446d-9ae1-71c26ae9bbfb-kube-api-access-b8c9p\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.135220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.133225 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-fbddf666f-qzmvf"] Jul 27 21:55:36.135675 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.135658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.138632 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.138613 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 21:55:36.138741 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.138676 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 21:55:36.138741 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.138683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-46wvs\"" Jul 27 21:55:36.138842 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.138747 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 21:55:36.139027 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.139013 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 21:55:36.139102 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.139087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 21:55:36.139139 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.139109 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 21:55:36.146767 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.146745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fbddf666f-qzmvf"] Jul 27 21:55:36.154325 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bzw\" (UniqueName: \"kubernetes.io/projected/0c78a3d5-3758-4f24-9833-955c51dd5063-kube-api-access-x5bzw\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.154443 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f988c022-2428-446d-9ae1-71c26ae9bbfb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.154443 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8c9p\" (UniqueName: \"kubernetes.io/projected/f988c022-2428-446d-9ae1-71c26ae9bbfb-kube-api-access-b8c9p\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.154443 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.154578 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.154578 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154497 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-default-certificate\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.154653 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.154585 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:36.154653 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-stats-auth\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.154740 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.154664 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:55:36.654642982 +0000 UTC m=+57.035714888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:36.154740 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.154686 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.155028 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.155010 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f988c022-2428-446d-9ae1-71c26ae9bbfb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.165437 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.165417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8c9p\" (UniqueName: \"kubernetes.io/projected/f988c022-2428-446d-9ae1-71c26ae9bbfb-kube-api-access-b8c9p\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.255191 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.255130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.255382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.255220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bzw\" (UniqueName: \"kubernetes.io/projected/0c78a3d5-3758-4f24-9833-955c51dd5063-kube-api-access-x5bzw\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.255382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.255274 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.255382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.255337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-default-certificate\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.255382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.255371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-stats-auth\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.255595 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.255382 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:36.255595 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.255467 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:36.755447491 +0000 UTC m=+57.136519393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:36.255595 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.255516 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:36.755489408 +0000 UTC m=+57.136561315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:36.256663 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.256635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:36.257973 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.257951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-stats-auth\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.258085 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.258046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-default-certificate\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.264525 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.264501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bzw\" (UniqueName: \"kubernetes.io/projected/0c78a3d5-3758-4f24-9833-955c51dd5063-kube-api-access-x5bzw\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.357560 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.357477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:36.658224 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.658189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:36.658388 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.658343 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:36.658442 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.658432 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:55:37.658416816 +0000 UTC m=+58.039488718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:36.758643 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.758607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.758802 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:36.758673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:36.758802 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.758785 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:37.75876293 +0000 UTC m=+58.139834832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:36.758883 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.758788 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:36.758883 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:36.758829 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:37.758820888 +0000 UTC m=+58.139892794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:37.666505 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:37.666463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:37.666991 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:37.666640 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:37.666991 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:37.666738 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:55:39.666714671 +0000 UTC m=+60.047786586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:37.766805 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:37.766769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:37.766964 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:37.766860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:37.766964 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:37.766934 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:39.766914698 +0000 UTC m=+60.147986603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:37.767038 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:37.766967 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:37.767038 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:37.767013 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:39.767000801 +0000 UTC m=+60.148072703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:39.682242 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:39.682204 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:39.682599 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:39.682364 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:39.682599 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:39.682441 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:55:43.682424467 +0000 UTC m=+64.063496374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:39.782770 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:39.782732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:39.782940 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:39.782804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:39.782940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:39.782905 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:43.782887523 +0000 UTC m=+64.163959429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:39.782940 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:39.782908 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:39.783048 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:39.782946 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:43.782939355 +0000 UTC m=+64.164011256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:40.264445 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.264415 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8q96t" Jul 27 21:55:40.422016 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.421975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:40.511518 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.511479 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5n9wf"] Jul 27 21:55:40.514084 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.514065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.515406 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.515330 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:40.516659 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.516639 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 21:55:40.516777 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.516761 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 21:55:40.516847 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.516768 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s5pg8\"" Jul 27 21:55:40.516904 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.516866 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 21:55:40.517070 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.517054 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 21:55:40.524870 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.524852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5n9wf"] Jul 27 21:55:40.587561 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.587525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.587735 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.587591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-crio-socket\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.587735 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.587621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-data-volume\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.587735 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.587638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664cl\" (UniqueName: \"kubernetes.io/projected/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-api-access-664cl\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.587735 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.587662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.602017 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.601991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:40.688395 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-crio-socket\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-data-volume\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664cl\" (UniqueName: \"kubernetes.io/projected/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-api-access-664cl\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-crio-socket\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:40.688632 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:40.688694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:41.18867516 +0000 UTC m=+61.569747064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : configmap "kube-rbac-proxy" not found Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:40.688760 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:40.688819 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:40.688820 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:41.188802561 +0000 UTC m=+61.569874484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:40.689140 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.688887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-data-volume\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:40.699671 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:40.699651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664cl\" (UniqueName: \"kubernetes.io/projected/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-api-access-664cl\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:41.192469 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:41.192433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:41.192638 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:41.192493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:41.192638 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:41.192551 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:55:41.192638 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:41.192616 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:42.19259998 +0000 UTC m=+62.573671885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : configmap "kube-rbac-proxy" not found Jul 27 21:55:41.192747 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:41.192638 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:41.192747 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:41.192685 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:42.192671441 +0000 UTC m=+62.573743350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:42.200752 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:42.200714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:42.201175 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:42.200777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:42.201175 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:42.200850 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:55:42.201175 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:42.200902 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:42.201175 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:42.200919 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:44.200902 +0000 UTC m=+64.581973906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : configmap "kube-rbac-proxy" not found Jul 27 21:55:42.201175 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:42.200959 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:44.200952336 +0000 UTC m=+64.582024237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:42.262575 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:42.262553 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzr5j_5343a980-96ea-4633-9a06-5511934c207f/dns-node-resolver/0.log" Jul 27 21:55:43.462941 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:43.462913 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5nx9_8f3b1d48-c616-4c79-8405-fe3c3f5cb03f/node-ca/0.log" Jul 27 21:55:43.712921 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:43.712885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:43.713090 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:43.713035 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:43.713141 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:43.713108 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:55:51.713092002 +0000 UTC m=+72.094163908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:43.813602 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:43.813567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:43.813742 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:43.813663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:43.813782 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:43.813741 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:51.8137222 +0000 UTC m=+72.194794101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:43.813782 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:43.813755 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:43.813847 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:43.813799 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:51.813787777 +0000 UTC m=+72.194859681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:44.062715 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:44.062632 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9kz5_e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6/serve-healthcheck-canary/0.log" Jul 27 21:55:44.217301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:44.217263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:44.217441 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:44.217328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:44.217441 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:44.217391 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:55:44.217526 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:44.217458 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:48.217442267 +0000 UTC m=+68.598514169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : configmap "kube-rbac-proxy" not found Jul 27 21:55:44.217526 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:44.217503 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:44.217595 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:44.217551 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:48.21753979 +0000 UTC m=+68.598611693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:45.830595 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.830540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:45.830595 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.830610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:45.831087 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.830654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:55:45.833210 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.833190 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:55:45.833276 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.833245 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:55:45.833367 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.833353 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:55:45.841054 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:45.841035 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 21:55:45.841102 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:45.841091 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs podName:ff07cd87-c25a-4f0c-a5e8-4b172891c63b nodeName:}" failed. No retries permitted until 2026-07-27 21:56:49.841075559 +0000 UTC m=+130.222147461 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs") pod "network-metrics-daemon-xzk78" (UID: "ff07cd87-c25a-4f0c-a5e8-4b172891c63b") : secret "metrics-daemon-secret" not found Jul 27 21:55:45.843139 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.843114 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:55:45.843366 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.843347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a851d07-33d7-4bcc-99e2-eb277ac875d4-original-pull-secret\") pod \"global-pull-secret-syncer-grl2x\" (UID: \"8a851d07-33d7-4bcc-99e2-eb277ac875d4\") " pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:45.853674 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.853645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9c2h\" (UniqueName: \"kubernetes.io/projected/1304f40d-41b1-46bd-9849-a756bb24911d-kube-api-access-d9c2h\") pod \"network-check-target-st8c7\" (UID: \"1304f40d-41b1-46bd-9849-a756bb24911d\") " pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:45.991585 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.991556 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ffd2\"" Jul 27 21:55:45.994449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.994434 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-grl2x" Jul 27 21:55:45.999997 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:45.999977 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:46.145242 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:46.145214 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-st8c7"] Jul 27 21:55:46.146142 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:55:46.146120 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1304f40d_41b1_46bd_9849_a756bb24911d.slice/crio-66842674290a3ae1f8abd324a2d6e30fd9b733e2e72f8a3315b27a04a017e137 WatchSource:0}: Error finding container 66842674290a3ae1f8abd324a2d6e30fd9b733e2e72f8a3315b27a04a017e137: Status 404 returned error can't find the container with id 66842674290a3ae1f8abd324a2d6e30fd9b733e2e72f8a3315b27a04a017e137 Jul 27 21:55:46.157195 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:46.157171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-grl2x"] Jul 27 21:55:46.160361 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:55:46.160336 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a851d07_33d7_4bcc_99e2_eb277ac875d4.slice/crio-8411776380a5da50cde85ae98ec1c6f09bb65cc8a351525dc1d77d0767cf9d0d WatchSource:0}: Error finding container 8411776380a5da50cde85ae98ec1c6f09bb65cc8a351525dc1d77d0767cf9d0d: Status 404 returned error can't find the container with id 8411776380a5da50cde85ae98ec1c6f09bb65cc8a351525dc1d77d0767cf9d0d Jul 27 21:55:46.355436 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:46.355353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-st8c7" event={"ID":"1304f40d-41b1-46bd-9849-a756bb24911d","Type":"ContainerStarted","Data":"66842674290a3ae1f8abd324a2d6e30fd9b733e2e72f8a3315b27a04a017e137"} Jul 27 21:55:46.356375 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:46.356354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-grl2x" event={"ID":"8a851d07-33d7-4bcc-99e2-eb277ac875d4","Type":"ContainerStarted","Data":"8411776380a5da50cde85ae98ec1c6f09bb65cc8a351525dc1d77d0767cf9d0d"} Jul 27 21:55:46.434968 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:46.434933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:55:46.435110 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:46.435082 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:55:46.435181 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:46.435168 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls podName:a6f1f903-b9f0-468c-9a33-4294fa33d347 nodeName:}" failed. No retries permitted until 2026-07-27 21:56:18.435132905 +0000 UTC m=+98.816204842 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls") pod "dns-default-hzf7h" (UID: "a6f1f903-b9f0-468c-9a33-4294fa33d347") : secret "dns-default-metrics-tls" not found Jul 27 21:55:48.249633 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:48.249592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:48.250078 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:48.249735 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:48.250078 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:48.249738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:48.250078 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:48.249804 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:56.24977936 +0000 UTC m=+76.630851262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:48.250078 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:48.249803 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:55:48.250078 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:48.249862 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:56.249844274 +0000 UTC m=+76.630916189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : configmap "kube-rbac-proxy" not found Jul 27 21:55:48.272834 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:48.272797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:48.337304 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:48.337264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:49.370124 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:49.370084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-st8c7" event={"ID":"1304f40d-41b1-46bd-9849-a756bb24911d","Type":"ContainerStarted","Data":"705c1c843cd6acf895f2ff6e34f33c944419133d9acebd8d77ccf3a27a07fb53"} Jul 27 21:55:49.370515 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:49.370201 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:55:49.385830 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:49.385775 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-st8c7" podStartSLOduration=66.371919495 podStartE2EDuration="1m9.385759747s" totalImagesPullingTime="3.013840252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:49.385015687 +0000 UTC m=+69.766087612" watchObservedRunningTime="2026-07-27 21:55:49.385759747 +0000 UTC m=+69.766831685" Jul 27 21:55:51.377308 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.377274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-grl2x" event={"ID":"8a851d07-33d7-4bcc-99e2-eb277ac875d4","Type":"ContainerStarted","Data":"2712d2a597a9b4530c01a46a3f7ee52075814e1a8e8f2e8204c8f2b504b5ec8b"} Jul 27 21:55:51.391745 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.391697 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-grl2x" podStartSLOduration=66.895501736 podStartE2EDuration="1m11.39168067s" totalImagesPullingTime="4.496178934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:51.390879899 +0000 UTC m=+71.771951815" watchObservedRunningTime="2026-07-27 21:55:51.39168067 +0000 UTC m=+71.772752593" Jul 27 21:55:51.621210 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.621179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:51.723974 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.723867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:51.779199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.779140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:55:51.779369 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:51.779297 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:51.779369 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:51.779363 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls podName:f988c022-2428-446d-9ae1-71c26ae9bbfb nodeName:}" failed. No retries permitted until 2026-07-27 21:56:07.779347476 +0000 UTC m=+88.160419377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-7gtpr" (UID: "f988c022-2428-446d-9ae1-71c26ae9bbfb") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:55:51.788908 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.788876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:51.879841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.879806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:51.879975 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.879869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:55:51.879975 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:51.879963 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:55:51.880044 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:51.880004 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:56:07.879987513 +0000 UTC m=+88.261059438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : configmap references non-existent config key: service-ca.crt Jul 27 21:55:51.880044 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:51.880021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs podName:0c78a3d5-3758-4f24-9833-955c51dd5063 nodeName:}" failed. No retries permitted until 2026-07-27 21:56:07.880013705 +0000 UTC m=+88.261085606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs") pod "router-default-fbddf666f-qzmvf" (UID: "0c78a3d5-3758-4f24-9833-955c51dd5063") : secret "router-metrics-certs-default" not found Jul 27 21:55:51.889351 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:51.889322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:52.054426 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:52.054340 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:52.157787 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:52.157747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:52.258530 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:52.258494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:52.363135 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:52.363098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:52.462727 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:52.462697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:53.169759 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:53.169717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:53.269385 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:53.269355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:53.781933 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:53.781898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:53.986791 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:53.986761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:54.057235 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:54.057133 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:55.048113 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:55.048077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:56.313419 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:56.313373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:56.313894 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:56.313446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:56.313894 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:56.313606 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:55:56.313894 ip-10-0-134-139 kubenswrapper[2559]: E0727 21:55:56.313687 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls podName:ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8 nodeName:}" failed. No retries permitted until 2026-07-27 21:56:12.313670321 +0000 UTC m=+92.694742223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5n9wf" (UID: "ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8") : secret "insights-runtime-extractor-tls" not found Jul 27 21:55:56.314009 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:56.313927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:55:57.954749 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:57.954710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.117503 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.117469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.220194 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.220071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.318893 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.318860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.400014 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.399981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.664319 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.664279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:58.727746 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:58.727718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:55:59.861202 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:55:59.861146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:00.058639 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:00.058604 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:00.257046 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:00.256953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:00.456243 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:00.456204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:00.697482 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:00.697451 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:01.665538 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:01.665501 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:01.767857 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:01.767830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:01.866171 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:01.866126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:02.264485 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:02.264450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:02.466522 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:02.466489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:02.539531 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:02.539456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:03.052228 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:03.052201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:03.685210 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:03.685180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:03.863706 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:03.863673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:04.061845 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:04.061752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:04.180058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:04.180024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:04.480294 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:04.480252 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:05.054723 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:05.054667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:05.455389 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:05.455349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:05.685626 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:05.685595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:06.046194 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:06.046139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:06.657120 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:06.657057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:06.945377 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:06.945270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:07.049696 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.049669 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:07.250685 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.250590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:07.653322 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.653285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:07.799444 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.799402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:56:07.801922 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.801903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f988c022-2428-446d-9ae1-71c26ae9bbfb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-7gtpr\" (UID: \"f988c022-2428-446d-9ae1-71c26ae9bbfb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:56:07.846134 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.846103 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xzw65\"" Jul 27 21:56:07.854540 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.854513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" Jul 27 21:56:07.899835 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.899805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:07.900023 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.899880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:07.900469 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.900441 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c78a3d5-3758-4f24-9833-955c51dd5063-service-ca-bundle\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:07.903942 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.902752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c78a3d5-3758-4f24-9833-955c51dd5063-metrics-certs\") pod \"router-default-fbddf666f-qzmvf\" (UID: \"0c78a3d5-3758-4f24-9833-955c51dd5063\") " pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:07.947214 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.947191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-46wvs\"" Jul 27 21:56:07.955238 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.955207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:07.981449 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:07.981420 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr"] Jul 27 21:56:07.984748 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:07.984719 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf988c022_2428_446d_9ae1_71c26ae9bbfb.slice/crio-50add4637171e122cf10f3cbba3ae6ebacbfcf4fd36cd84edd3824ea5496fd25 WatchSource:0}: Error finding container 50add4637171e122cf10f3cbba3ae6ebacbfcf4fd36cd84edd3824ea5496fd25: Status 404 returned error can't find the container with id 50add4637171e122cf10f3cbba3ae6ebacbfcf4fd36cd84edd3824ea5496fd25 Jul 27 21:56:08.048571 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.048543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:08.087070 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.087040 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fbddf666f-qzmvf"] Jul 27 21:56:08.090069 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:08.090048 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c78a3d5_3758_4f24_9833_955c51dd5063.slice/crio-a0f1908b986a49ac449b556c742858a61ce1308ea0db71d096a3683b74a66b43 WatchSource:0}: Error finding container a0f1908b986a49ac449b556c742858a61ce1308ea0db71d096a3683b74a66b43: Status 404 returned error can't find the container with id a0f1908b986a49ac449b556c742858a61ce1308ea0db71d096a3683b74a66b43 Jul 27 21:56:08.419374 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.419337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fbddf666f-qzmvf" event={"ID":"0c78a3d5-3758-4f24-9833-955c51dd5063","Type":"ContainerStarted","Data":"c59a03b76761dfaca05efd26d7734c770767cc3637cba38dba9fab69348f26a4"} Jul 27 21:56:08.419562 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.419416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fbddf666f-qzmvf" event={"ID":"0c78a3d5-3758-4f24-9833-955c51dd5063","Type":"ContainerStarted","Data":"a0f1908b986a49ac449b556c742858a61ce1308ea0db71d096a3683b74a66b43"} Jul 27 21:56:08.420493 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.420459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" event={"ID":"f988c022-2428-446d-9ae1-71c26ae9bbfb","Type":"ContainerStarted","Data":"50add4637171e122cf10f3cbba3ae6ebacbfcf4fd36cd84edd3824ea5496fd25"} Jul 27 21:56:08.438499 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.438437 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-fbddf666f-qzmvf" podStartSLOduration=32.438422837 podStartE2EDuration="32.438422837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:08.437813653 +0000 UTC m=+88.818885577" watchObservedRunningTime="2026-07-27 21:56:08.438422837 +0000 UTC m=+88.819494761" Jul 27 21:56:08.453050 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.453017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:08.650857 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.650817 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:08.956373 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.956333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:08.959334 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:08.959295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:09.423828 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:09.423796 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:09.425208 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:09.425187 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-fbddf666f-qzmvf" Jul 27 21:56:10.428530 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.428442 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" event={"ID":"f988c022-2428-446d-9ae1-71c26ae9bbfb","Type":"ContainerStarted","Data":"7426257ffe26c59c31164deab5ddf83d2aed78721c73c34de2a7b750347ca822"} Jul 27 21:56:10.448007 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.447956 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" podStartSLOduration=32.277257696 podStartE2EDuration="34.447940292s" totalImagesPullingTime="2.170682596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:10.446593463 +0000 UTC m=+90.827665387" watchObservedRunningTime="2026-07-27 21:56:10.447940292 +0000 UTC m=+90.829012216" Jul 27 21:56:10.464277 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.464243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:10.578727 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.578675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:10.671290 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.671252 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:10.798573 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:10.798459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:12.335266 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:12.335218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:56:12.337635 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:12.337613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5n9wf\" (UID: \"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8\") " pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:56:12.623366 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:12.623341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5n9wf" Jul 27 21:56:12.748094 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:12.748064 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5n9wf"] Jul 27 21:56:12.751441 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:12.751417 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae19c50f_c24b_4bb8_a8ec_cf8bc1ce14e8.slice/crio-9d60fcd4891e779709d59318aa5c7c4c15799f3fd4ebd9041c975a2afdc9f21f WatchSource:0}: Error finding container 9d60fcd4891e779709d59318aa5c7c4c15799f3fd4ebd9041c975a2afdc9f21f: Status 404 returned error can't find the container with id 9d60fcd4891e779709d59318aa5c7c4c15799f3fd4ebd9041c975a2afdc9f21f Jul 27 21:56:13.439639 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.439542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5n9wf" event={"ID":"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8","Type":"ContainerStarted","Data":"9d60fcd4891e779709d59318aa5c7c4c15799f3fd4ebd9041c975a2afdc9f21f"} Jul 27 21:56:13.674103 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.674069 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs"] Jul 27 21:56:13.678404 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.678379 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.681620 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.681596 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 21:56:13.682590 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.682559 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 21:56:13.682740 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.682722 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 21:56:13.685030 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.685008 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 27 21:56:13.686481 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.686456 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm"] Jul 27 21:56:13.689886 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.689818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.694181 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.694136 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 21:56:13.694494 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.694375 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 21:56:13.694494 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.694400 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 21:56:13.701164 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.701124 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 21:56:13.701458 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.701379 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs"] Jul 27 21:56:13.724232 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.724175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm"] Jul 27 21:56:13.746014 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.745976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-tmp\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.746195 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-klusterlet-config\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.746195 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.746195 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmgw\" (UniqueName: \"kubernetes.io/projected/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-kube-api-access-hhmgw\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.746347 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vc8s\" (UniqueName: \"kubernetes.io/projected/82e0166a-452e-4952-a24f-898d8cb897e3-kube-api-access-4vc8s\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.746347 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e0166a-452e-4952-a24f-898d8cb897e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.746347 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746304 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.746445 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.746445 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.746371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.776201 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.776145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:13.847054 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-klusterlet-config\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.847265 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847265 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmgw\" (UniqueName: \"kubernetes.io/projected/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-kube-api-access-hhmgw\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.847265 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vc8s\" (UniqueName: \"kubernetes.io/projected/82e0166a-452e-4952-a24f-898d8cb897e3-kube-api-access-4vc8s\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847265 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e0166a-452e-4952-a24f-898d8cb897e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847265 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847537 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847537 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.847537 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-tmp\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.847787 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.847726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-tmp\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.848417 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.848381 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e0166a-452e-4952-a24f-898d8cb897e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.850496 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.850474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-klusterlet-config\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.850608 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.850477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.850961 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.850901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-ca\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.851034 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.850976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-hub\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.851767 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.851744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/82e0166a-452e-4952-a24f-898d8cb897e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.869133 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.869088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vc8s\" (UniqueName: \"kubernetes.io/projected/82e0166a-452e-4952-a24f-898d8cb897e3-kube-api-access-4vc8s\") pod \"cluster-proxy-proxy-agent-7dd4488c59-z9lcm\" (UID: \"82e0166a-452e-4952-a24f-898d8cb897e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:13.870142 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.870122 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmgw\" (UniqueName: \"kubernetes.io/projected/0bfa3a40-0f7a-41d8-bbe1-97d6c593c333-kube-api-access-hhmgw\") pod \"klusterlet-addon-workmgr-68fcbd5cd4-r2kxs\" (UID: \"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:13.908850 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.908756 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:13.984342 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.984259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:13.993220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:13.993193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:14.012836 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:14.012807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" Jul 27 21:56:14.154528 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:14.154500 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs"] Jul 27 21:56:14.158038 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:14.158007 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfa3a40_0f7a_41d8_bbe1_97d6c593c333.slice/crio-d6852d4b104573575b744c66116a6bbbab7376b6f68b3b9f2f8d06730f9fd358 WatchSource:0}: Error finding container d6852d4b104573575b744c66116a6bbbab7376b6f68b3b9f2f8d06730f9fd358: Status 404 returned error can't find the container with id d6852d4b104573575b744c66116a6bbbab7376b6f68b3b9f2f8d06730f9fd358 Jul 27 21:56:14.174918 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:14.174336 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm"] Jul 27 21:56:14.177725 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:14.177695 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e0166a_452e_4952_a24f_898d8cb897e3.slice/crio-4424c90ad04d12b567cacd17562d5155b80d43d4325ca752307010d3025b78f3 WatchSource:0}: Error finding container 4424c90ad04d12b567cacd17562d5155b80d43d4325ca752307010d3025b78f3: Status 404 returned error can't find the container with id 4424c90ad04d12b567cacd17562d5155b80d43d4325ca752307010d3025b78f3 Jul 27 21:56:14.444295 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:14.444254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" event={"ID":"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333","Type":"ContainerStarted","Data":"d6852d4b104573575b744c66116a6bbbab7376b6f68b3b9f2f8d06730f9fd358"} Jul 27 21:56:14.445555 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:14.445522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" event={"ID":"82e0166a-452e-4952-a24f-898d8cb897e3","Type":"ContainerStarted","Data":"4424c90ad04d12b567cacd17562d5155b80d43d4325ca752307010d3025b78f3"} Jul 27 21:56:18.365407 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.365369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:18.465692 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.465644 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:18.483712 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.483672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:56:18.486495 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.486473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6f1f903-b9f0-468c-9a33-4294fa33d347-metrics-tls\") pod \"dns-default-hzf7h\" (UID: \"a6f1f903-b9f0-468c-9a33-4294fa33d347\") " pod="openshift-dns/dns-default-hzf7h" Jul 27 21:56:18.715937 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.715854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:18.788880 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.788848 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jz2cl\"" Jul 27 21:56:18.794198 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.794169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:18.796881 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.796855 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzf7h" Jul 27 21:56:18.866984 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:18.866940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:20.376447 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:20.376420 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-st8c7" Jul 27 21:56:21.740046 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:21.740024 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzf7h"] Jul 27 21:56:21.743343 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:21.743316 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f1f903_b9f0_468c_9a33_4294fa33d347.slice/crio-1043a8ef34b252d7e5c137bf74218426363ff1b8182210207661aaeea906135a WatchSource:0}: Error finding container 1043a8ef34b252d7e5c137bf74218426363ff1b8182210207661aaeea906135a: Status 404 returned error can't find the container with id 1043a8ef34b252d7e5c137bf74218426363ff1b8182210207661aaeea906135a Jul 27 21:56:22.473824 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.473789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5n9wf" event={"ID":"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8","Type":"ContainerStarted","Data":"2b70f14a7b3a6fd311dffdf0bfd29abaa4e5a5fd69139643b2d447554784d18a"} Jul 27 21:56:22.475748 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.475716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" event={"ID":"0bfa3a40-0f7a-41d8-bbe1-97d6c593c333","Type":"ContainerStarted","Data":"4e9bd61c86a90ef00dcee9a5348b5528a825c8244c56dd882bbfdf82403734f3"} Jul 27 21:56:22.476210 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.476187 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:22.478220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.478173 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" Jul 27 21:56:22.480058 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.480025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" event={"ID":"82e0166a-452e-4952-a24f-898d8cb897e3","Type":"ContainerStarted","Data":"27eeaf1b1c6eca4c43c7c3877c9ed0b7ef5c4a83035ba490128413c20c776296"} Jul 27 21:56:22.480176 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.480062 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" event={"ID":"82e0166a-452e-4952-a24f-898d8cb897e3","Type":"ContainerStarted","Data":"e99fdaf86b9e7f7e243046cd621835d811ea9c9dc8b848b5e5c13ecce62bd79d"} Jul 27 21:56:22.480176 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.480074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" event={"ID":"82e0166a-452e-4952-a24f-898d8cb897e3","Type":"ContainerStarted","Data":"5d864dbee209be3b9dca6515fd63acc411f00083ea38837df1f1b1ebd9fed466"} Jul 27 21:56:22.481587 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.481534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzf7h" event={"ID":"a6f1f903-b9f0-468c-9a33-4294fa33d347","Type":"ContainerStarted","Data":"1043a8ef34b252d7e5c137bf74218426363ff1b8182210207661aaeea906135a"} Jul 27 21:56:22.492478 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.492434 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68fcbd5cd4-r2kxs" podStartSLOduration=2.089907517 podStartE2EDuration="9.492421807s" totalImagesPullingTime="7.40251429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:22.491671507 +0000 UTC m=+102.872743431" watchObservedRunningTime="2026-07-27 21:56:22.492421807 +0000 UTC m=+102.873493731" Jul 27 21:56:22.530392 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:22.529391 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd4488c59-z9lcm" podStartSLOduration=2.129718321 podStartE2EDuration="9.529367463s" totalImagesPullingTime="7.399649142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:22.528498691 +0000 UTC m=+102.909570632" watchObservedRunningTime="2026-07-27 21:56:22.529367463 +0000 UTC m=+102.910439388" Jul 27 21:56:23.009798 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:23.009749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:25.836047 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:25.835965 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:25.936789 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:25.936716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:26.499202 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.499142 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzf7h" event={"ID":"a6f1f903-b9f0-468c-9a33-4294fa33d347","Type":"ContainerStarted","Data":"15606fc4b303e86f06e5a7fe038caec5924f8c39de8d2d7265157d63789fd373"} Jul 27 21:56:26.499389 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.499234 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzf7h" event={"ID":"a6f1f903-b9f0-468c-9a33-4294fa33d347","Type":"ContainerStarted","Data":"17c8abc610267c018e9b7642206e277e2a690841cfb96e103132d642d9c7ce59"} Jul 27 21:56:26.499389 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.499272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzf7h" Jul 27 21:56:26.500991 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.500963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5n9wf" event={"ID":"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8","Type":"ContainerStarted","Data":"ded3c4d83103def5891c468b1751985ad672f0a225a4b65d27f69e91937f6c56"} Jul 27 21:56:26.502388 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.502369 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 21:56:26.502512 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.502401 2559 generic.go:356] "Generic (PLEG): container finished" podID="f988c022-2428-446d-9ae1-71c26ae9bbfb" containerID="7426257ffe26c59c31164deab5ddf83d2aed78721c73c34de2a7b750347ca822" exitCode=2 Jul 27 21:56:26.502512 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.502432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" event={"ID":"f988c022-2428-446d-9ae1-71c26ae9bbfb","Type":"ContainerDied","Data":"7426257ffe26c59c31164deab5ddf83d2aed78721c73c34de2a7b750347ca822"} Jul 27 21:56:26.502712 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.502698 2559 scope.go:117] "RemoveContainer" containerID="7426257ffe26c59c31164deab5ddf83d2aed78721c73c34de2a7b750347ca822" Jul 27 21:56:26.516780 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:26.516689 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hzf7h" podStartSLOduration=68.167501511 podStartE2EDuration="1m12.516669317s" totalImagesPullingTime="4.349167806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:26.516399685 +0000 UTC m=+106.897471610" watchObservedRunningTime="2026-07-27 21:56:26.516669317 +0000 UTC m=+106.897741248" Jul 27 21:56:27.412299 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:27.412244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:27.508862 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:27.508832 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 21:56:27.509040 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:27.508983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-7gtpr" event={"ID":"f988c022-2428-446d-9ae1-71c26ae9bbfb","Type":"ContainerStarted","Data":"4d5f2a938fd073b37cbb8a25c640a9cd342d5dd3613faabcb828842e8258f241"} Jul 27 21:56:27.617382 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:27.617353 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:28.697528 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:28.697486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:28.758187 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:28.758140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.159692 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.159650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.200443 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.200413 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b7n6c"] Jul 27 21:56:30.203727 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.203707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.207071 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.207044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 21:56:30.207071 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.207061 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 21:56:30.207287 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.207133 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 21:56:30.207287 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.207133 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jqbc5\"" Jul 27 21:56:30.207775 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.207761 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 21:56:30.259399 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.259360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.283050 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-metrics-client-ca\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283050 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klznc\" (UniqueName: \"kubernetes.io/projected/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-kube-api-access-klznc\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283307 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283307 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-root\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283307 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-textfile\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283307 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283274 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283476 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-wtmp\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283476 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-tls\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.283476 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.283427 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-sys\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.357266 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.357233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.383753 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-sys\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.383753 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-metrics-client-ca\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klznc\" (UniqueName: \"kubernetes.io/projected/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-kube-api-access-klznc\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-root\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383841 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-textfile\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-root\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-wtmp\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383868 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-sys\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.383943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-tls\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384483 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.384126 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-wtmp\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384483 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.384337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-textfile\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384584 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.384526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.384584 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.384532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-metrics-client-ca\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.386483 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.386461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.386599 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.386465 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-node-exporter-tls\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.392726 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.392700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klznc\" (UniqueName: \"kubernetes.io/projected/dc2b703a-a49d-4b0e-a35e-ac6239d4ae39-kube-api-access-klznc\") pod \"node-exporter-b7n6c\" (UID: \"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39\") " pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.457394 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.457293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.512480 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.512444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b7n6c" Jul 27 21:56:30.522332 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.522304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5n9wf" event={"ID":"ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8","Type":"ContainerStarted","Data":"dcc58a2bb9f5985f4b74c347afaf4806413453505c55bda5c1acc9a0f5432bd6"} Jul 27 21:56:30.524730 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:30.524709 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2b703a_a49d_4b0e_a35e_ac6239d4ae39.slice/crio-75f8e6806abb486db0e99df243a86ef60cee4d598b7bcf95970dc25cbded97ed WatchSource:0}: Error finding container 75f8e6806abb486db0e99df243a86ef60cee4d598b7bcf95970dc25cbded97ed: Status 404 returned error can't find the container with id 75f8e6806abb486db0e99df243a86ef60cee4d598b7bcf95970dc25cbded97ed Jul 27 21:56:30.555581 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.555526 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5n9wf" podStartSLOduration=33.797419769 podStartE2EDuration="50.555508748s" totalImagesPullingTime="16.758088979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:55:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:30.554724448 +0000 UTC m=+110.935796373" watchObservedRunningTime="2026-07-27 21:56:30.555508748 +0000 UTC m=+110.936580672" Jul 27 21:56:30.560322 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.560287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.661425 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.661393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.758708 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.758616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:30.860289 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:30.860256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:31.154841 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:31.154788 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:31.311848 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:31.311815 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:31.383726 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:31.383683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:31.527444 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:31.527368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7n6c" event={"ID":"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39","Type":"ContainerStarted","Data":"75f8e6806abb486db0e99df243a86ef60cee4d598b7bcf95970dc25cbded97ed"} Jul 27 21:56:32.531870 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:32.531831 2559 generic.go:356] "Generic (PLEG): container finished" podID="dc2b703a-a49d-4b0e-a35e-ac6239d4ae39" containerID="74903ff3d66ede365fb661f52c07e450022db8c68be3f198161de12b7946c51e" exitCode=0 Jul 27 21:56:32.532286 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:32.531897 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7n6c" event={"ID":"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39","Type":"ContainerDied","Data":"74903ff3d66ede365fb661f52c07e450022db8c68be3f198161de12b7946c51e"} Jul 27 21:56:33.089251 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.089209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:33.171830 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.171791 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:33.248445 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.248411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:33.458749 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.458720 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:33.537131 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.537097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7n6c" event={"ID":"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39","Type":"ContainerStarted","Data":"021a640db4aeddba8b7b3b665d2d76d17e95f309b85f128432d8033cd0d36eab"} Jul 27 21:56:33.537605 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.537213 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7n6c" event={"ID":"dc2b703a-a49d-4b0e-a35e-ac6239d4ae39","Type":"ContainerStarted","Data":"58e418ef2fbfc93508638184def256dd1aac950cf8affd6c57eff0040bb4bd54"} Jul 27 21:56:33.558048 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:33.558003 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b7n6c" podStartSLOduration=2.473221944 podStartE2EDuration="3.55798734s" totalImagesPullingTime="1.014517888s" totalInitContainerRuntime="70.247508ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:33.556873544 +0000 UTC m=+113.937945468" watchObservedRunningTime="2026-07-27 21:56:33.55798734 +0000 UTC m=+113.939059266" Jul 27 21:56:34.952902 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:34.952846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:35.662054 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:35.662025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:36.343896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:36.343862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:36.511498 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:36.511466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzf7h" Jul 27 21:56:39.114161 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:39.114122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:39.170318 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:39.170286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:43.715326 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:43.715294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:43.820782 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:43.820753 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.168640 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.168599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.271316 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.271277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.448885 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.448794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.620788 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.620759 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.679610 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.679579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.843386 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.843293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:44.904966 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:44.904937 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:45.700621 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:45.700590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:45.757045 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:45.757007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:45.839787 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:45.839755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:45.982358 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:45.982265 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:46.092014 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:46.091983 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:46.150196 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:46.150166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:46.226710 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:46.226682 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:46.964544 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:46.964517 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzf7h_a6f1f903-b9f0-468c-9a33-4294fa33d347/dns/0.log" Jul 27 21:56:47.148980 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:47.148945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:47.166207 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:47.166128 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzf7h_a6f1f903-b9f0-468c-9a33-4294fa33d347/kube-rbac-proxy/0.log" Jul 27 21:56:47.249259 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:47.249173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:47.764040 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:47.764010 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzr5j_5343a980-96ea-4633-9a06-5511934c207f/dns-node-resolver/0.log" Jul 27 21:56:48.456731 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:48.456704 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:48.512538 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:48.512512 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:49.164523 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:49.164495 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5nx9_8f3b1d48-c616-4c79-8405-fe3c3f5cb03f/node-ca/0.log" Jul 27 21:56:49.365767 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:49.365740 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fbddf666f-qzmvf_0c78a3d5-3758-4f24-9833-955c51dd5063/router/0.log" Jul 27 21:56:49.764989 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:49.764949 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9kz5_e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6/serve-healthcheck-canary/0.log" Jul 27 21:56:49.928750 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:49.928713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:56:49.931085 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:49.931061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff07cd87-c25a-4f0c-a5e8-4b172891c63b-metrics-certs\") pod \"network-metrics-daemon-xzk78\" (UID: \"ff07cd87-c25a-4f0c-a5e8-4b172891c63b\") " pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:56:50.201496 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:50.201464 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tm84h\"" Jul 27 21:56:50.209482 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:50.209455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzk78" Jul 27 21:56:50.520687 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:50.520653 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzk78"] Jul 27 21:56:50.525259 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:56:50.525228 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff07cd87_c25a_4f0c_a5e8_4b172891c63b.slice/crio-934b6ef622de32c3e8836b4dc2044c2d5c930ed28545c79695eb7d3c1503f7ac WatchSource:0}: Error finding container 934b6ef622de32c3e8836b4dc2044c2d5c930ed28545c79695eb7d3c1503f7ac: Status 404 returned error can't find the container with id 934b6ef622de32c3e8836b4dc2044c2d5c930ed28545c79695eb7d3c1503f7ac Jul 27 21:56:50.592911 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:50.592878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzk78" event={"ID":"ff07cd87-c25a-4f0c-a5e8-4b172891c63b","Type":"ContainerStarted","Data":"934b6ef622de32c3e8836b4dc2044c2d5c930ed28545c79695eb7d3c1503f7ac"} Jul 27 21:56:51.457101 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:51.457059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:51.556017 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:51.555973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:51.657683 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:51.657653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:52.602527 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:52.602442 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzk78" event={"ID":"ff07cd87-c25a-4f0c-a5e8-4b172891c63b","Type":"ContainerStarted","Data":"b0eb58e19cb1b27cd7481f2ba14f3f533a8ecf7dd371bff5254c73aba172f49f"} Jul 27 21:56:52.602527 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:52.602490 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzk78" event={"ID":"ff07cd87-c25a-4f0c-a5e8-4b172891c63b","Type":"ContainerStarted","Data":"1c21256a5914a982313b36f05fc0341ae7fab41036e3a8b0d623451abcdbac50"} Jul 27 21:56:52.619287 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:52.619232 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xzk78" podStartSLOduration=131.023266852 podStartE2EDuration="2m12.61921634s" totalImagesPullingTime="1.595949488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:52.619021673 +0000 UTC m=+133.000093599" watchObservedRunningTime="2026-07-27 21:56:52.61921634 +0000 UTC m=+133.000288263" Jul 27 21:56:52.722792 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:52.722751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:52.824355 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:52.824313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:54.643609 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:54.643576 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:54.744756 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:54.744723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:56.258765 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:56.258730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:56.323000 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:56.322968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:58.700572 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:58.700536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:58.760730 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:56:58.760697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:00.446496 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:00.446442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:03.158930 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:03.158896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:03.243546 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:03.243271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:06.045938 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:06.045901 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:06.108418 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:06.108375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:09.256478 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:09.256444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:09.311729 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:09.311701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:14.268220 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:14.268187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:14.328553 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:14.328522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:14.393942 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:14.393905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:20.629037 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:20.629000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:20.693501 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:20.693467 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:28.692312 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:28.692259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:28.744322 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:28.744293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:30.199433 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:30.199395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:30.299323 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:30.299290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:43.704519 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:43.704429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:43.767344 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:43.767311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:46.244894 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:46.244863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:46.307396 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:46.307369 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:49.294309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:49.294278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:57:59.352784 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:57:59.352749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:01.509778 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:01.509740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:01.661977 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:01.661946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:13.711110 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:13.711075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:13.809680 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:13.809622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:14.538401 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:14.538362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:14.638458 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:14.638418 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:14.740904 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:14.740867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:15.225868 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:15.225830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:15.296910 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:15.296875 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:16.349612 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:16.349581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:16.450102 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:16.450053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:25.519565 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:25.519534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:25.620542 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:25.620510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:25.722321 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:25.722269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:51.048291 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.048249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:51.138816 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.138780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:51.798708 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.798677 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz"] Jul 27 21:58:51.801896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.801881 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.804331 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.804310 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:58:51.805079 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.805063 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:58:51.805176 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.805070 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x9f8r\"" Jul 27 21:58:51.811989 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.811964 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz"] Jul 27 21:58:51.894961 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.894930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.894961 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.894962 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvxv\" (UniqueName: \"kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.895234 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.895007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.995572 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.995544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.995748 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.995583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.995748 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.995610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvxv\" (UniqueName: \"kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.995940 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.995921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:51.996017 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:51.995970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:52.003728 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:52.003695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvxv\" (UniqueName: \"kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:52.111164 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:52.111125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:58:52.240212 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:52.240176 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz"] Jul 27 21:58:52.243055 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:58:52.243030 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1d3911_41c2_41e9_a338_65d024d3c23c.slice/crio-261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec WatchSource:0}: Error finding container 261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec: Status 404 returned error can't find the container with id 261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec Jul 27 21:58:52.996710 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:52.996671 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" event={"ID":"ca1d3911-41c2-41e9-a338-65d024d3c23c","Type":"ContainerStarted","Data":"261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec"} Jul 27 21:58:54.684879 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:54.684834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:58:59.021867 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:59.021833 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerID="6cd6d7f5d87165a0b65f8b3684e8da9795ccba9f042c285c5e24f9fdcd012511" exitCode=0 Jul 27 21:58:59.022306 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:58:59.021920 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" event={"ID":"ca1d3911-41c2-41e9-a338-65d024d3c23c","Type":"ContainerDied","Data":"6cd6d7f5d87165a0b65f8b3684e8da9795ccba9f042c285c5e24f9fdcd012511"} Jul 27 21:59:02.035113 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:02.035074 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerID="806bac202b3395a0da6b295c32c5348c450af38f1f7f401a3606f5a200ada039" exitCode=0 Jul 27 21:59:02.035483 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:02.035166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" event={"ID":"ca1d3911-41c2-41e9-a338-65d024d3c23c","Type":"ContainerDied","Data":"806bac202b3395a0da6b295c32c5348c450af38f1f7f401a3606f5a200ada039"} Jul 27 21:59:09.064205 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:09.064139 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerID="bba1c9631d1be18bba768e43c2b49dbb6bed3c48550420cdd76faaa5686da2bd" exitCode=0 Jul 27 21:59:09.064205 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:09.064190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" event={"ID":"ca1d3911-41c2-41e9-a338-65d024d3c23c","Type":"ContainerDied","Data":"bba1c9631d1be18bba768e43c2b49dbb6bed3c48550420cdd76faaa5686da2bd"} Jul 27 21:59:10.186649 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.186621 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:59:10.250314 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.250284 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle\") pod \"ca1d3911-41c2-41e9-a338-65d024d3c23c\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " Jul 27 21:59:10.250488 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.250344 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util\") pod \"ca1d3911-41c2-41e9-a338-65d024d3c23c\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " Jul 27 21:59:10.250488 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.250394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnvxv\" (UniqueName: \"kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv\") pod \"ca1d3911-41c2-41e9-a338-65d024d3c23c\" (UID: \"ca1d3911-41c2-41e9-a338-65d024d3c23c\") " Jul 27 21:59:10.251015 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.250984 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle" (OuterVolumeSpecName: "bundle") pod "ca1d3911-41c2-41e9-a338-65d024d3c23c" (UID: "ca1d3911-41c2-41e9-a338-65d024d3c23c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:10.252668 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.252635 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv" (OuterVolumeSpecName: "kube-api-access-dnvxv") pod "ca1d3911-41c2-41e9-a338-65d024d3c23c" (UID: "ca1d3911-41c2-41e9-a338-65d024d3c23c"). InnerVolumeSpecName "kube-api-access-dnvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:59:10.254582 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.254560 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util" (OuterVolumeSpecName: "util") pod "ca1d3911-41c2-41e9-a338-65d024d3c23c" (UID: "ca1d3911-41c2-41e9-a338-65d024d3c23c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:10.350941 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.350854 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvxv\" (UniqueName: \"kubernetes.io/projected/ca1d3911-41c2-41e9-a338-65d024d3c23c-kube-api-access-dnvxv\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:10.350941 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.350882 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:10.350941 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:10.350892 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca1d3911-41c2-41e9-a338-65d024d3c23c-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:11.072317 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:11.072285 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" event={"ID":"ca1d3911-41c2-41e9-a338-65d024d3c23c","Type":"ContainerDied","Data":"261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec"} Jul 27 21:59:11.072486 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:11.072331 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261ab226e751de07c3f44d75aecf1fc3801747972c359ba7f78596a04cac97ec" Jul 27 21:59:11.072486 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:11.072292 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754067thz" Jul 27 21:59:12.998773 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:12.998743 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:14.367805 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.367770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:14.973164 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973115 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr"] Jul 27 21:59:14.973381 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973369 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="pull" Jul 27 21:59:14.973435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973383 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="pull" Jul 27 21:59:14.973435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973392 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="util" Jul 27 21:59:14.973435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973397 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="util" Jul 27 21:59:14.973435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973407 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="extract" Jul 27 21:59:14.973435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973413 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="extract" Jul 27 21:59:14.973599 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.973457 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1d3911-41c2-41e9-a338-65d024d3c23c" containerName="extract" Jul 27 21:59:14.975217 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.975199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:14.977647 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.977625 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 21:59:14.977753 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.977673 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mprxg\"" Jul 27 21:59:14.978925 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.978909 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:59:14.993008 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:14.992984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr"] Jul 27 21:59:15.087093 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.087064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q45x\" (UniqueName: \"kubernetes.io/projected/f398370a-0773-4288-b926-2b113a17c481-kube-api-access-7q45x\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.087277 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.087103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f398370a-0773-4288-b926-2b113a17c481-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.187746 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.187706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q45x\" (UniqueName: \"kubernetes.io/projected/f398370a-0773-4288-b926-2b113a17c481-kube-api-access-7q45x\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.187950 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.187895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f398370a-0773-4288-b926-2b113a17c481-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.188331 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.188311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f398370a-0773-4288-b926-2b113a17c481-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.198528 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.198498 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q45x\" (UniqueName: \"kubernetes.io/projected/f398370a-0773-4288-b926-2b113a17c481-kube-api-access-7q45x\") pod \"cert-manager-operator-controller-manager-55c444bf86-kzrrr\" (UID: \"f398370a-0773-4288-b926-2b113a17c481\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.283749 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.283662 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" Jul 27 21:59:15.413222 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:15.413184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr"] Jul 27 21:59:15.417739 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:59:15.417712 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf398370a_0773_4288_b926_2b113a17c481.slice/crio-f724d0a02cda3f1bf184c232a1f5abfee77d34f8245755574c527e88c2cbb910 WatchSource:0}: Error finding container f724d0a02cda3f1bf184c232a1f5abfee77d34f8245755574c527e88c2cbb910: Status 404 returned error can't find the container with id f724d0a02cda3f1bf184c232a1f5abfee77d34f8245755574c527e88c2cbb910 Jul 27 21:59:16.089617 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:16.089579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" event={"ID":"f398370a-0773-4288-b926-2b113a17c481","Type":"ContainerStarted","Data":"f724d0a02cda3f1bf184c232a1f5abfee77d34f8245755574c527e88c2cbb910"} Jul 27 21:59:16.610216 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:16.610181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:16.706502 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:16.706461 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.059864 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.059826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.098857 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.098829 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" event={"ID":"f398370a-0773-4288-b926-2b113a17c481","Type":"ContainerStarted","Data":"8d21799b77d94aaa5a0dcc44506e85011217818d6f2fa13a38e3f85e4765be16"} Jul 27 21:59:18.121645 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.121600 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kzrrr" podStartSLOduration=2.288790628 podStartE2EDuration="4.121582777s" totalImagesPullingTime="1.832792149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:59:18.121191527 +0000 UTC m=+278.502263449" watchObservedRunningTime="2026-07-27 21:59:18.121582777 +0000 UTC m=+278.502654699" Jul 27 21:59:18.129309 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.129279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.490415 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.490383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:18.598898 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:18.598860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:19.630261 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.630228 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw"] Jul 27 21:59:19.632350 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.632335 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.634654 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.634634 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:59:19.634654 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.634644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:59:19.634795 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.634650 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x9f8r\"" Jul 27 21:59:19.641683 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.641660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw"] Jul 27 21:59:19.720112 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.720080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.720296 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.720124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.720296 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.720165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwl6\" (UniqueName: \"kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.821515 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.821484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.821690 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.821534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.821690 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.821553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwl6\" (UniqueName: \"kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.821905 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.821886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.821942 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.821899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.830119 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.830095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwl6\" (UniqueName: \"kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:19.940801 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:19.940722 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:20.075099 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:20.075068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:20.075422 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:20.075399 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw"] Jul 27 21:59:20.079007 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:59:20.078981 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df8eef8_c9a1_462a_a3de_7acb29f84f86.slice/crio-c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318 WatchSource:0}: Error finding container c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318: Status 404 returned error can't find the container with id c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318 Jul 27 21:59:20.105722 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:20.105699 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" event={"ID":"8df8eef8-c9a1-462a-a3de-7acb29f84f86","Type":"ContainerStarted","Data":"c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318"} Jul 27 21:59:20.464896 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:20.464867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:21.110915 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:21.110884 2559 generic.go:356] "Generic (PLEG): container finished" podID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerID="6e82fc46c86ee29ec6c675a4e016d99820a0966b91228eca033fb1e4ea8428ea" exitCode=0 Jul 27 21:59:21.111324 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:21.110947 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" event={"ID":"8df8eef8-c9a1-462a-a3de-7acb29f84f86","Type":"ContainerDied","Data":"6e82fc46c86ee29ec6c675a4e016d99820a0966b91228eca033fb1e4ea8428ea"} Jul 27 21:59:23.410627 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:23.410594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:23.470501 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:23.470466 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:24.123450 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:24.123413 2559 generic.go:356] "Generic (PLEG): container finished" podID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerID="815a82df3b90731ec1ba78d7a62f032a46f0983deae4bab5903bac7b03b8d2d5" exitCode=0 Jul 27 21:59:24.123450 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:24.123453 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" event={"ID":"8df8eef8-c9a1-462a-a3de-7acb29f84f86","Type":"ContainerDied","Data":"815a82df3b90731ec1ba78d7a62f032a46f0983deae4bab5903bac7b03b8d2d5"} Jul 27 21:59:25.129278 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:25.129242 2559 generic.go:356] "Generic (PLEG): container finished" podID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerID="d03908df57bc643dbe646b1713afa8cff2075b3a3d5fd362429e654e9fb396aa" exitCode=0 Jul 27 21:59:25.129679 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:25.129294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" event={"ID":"8df8eef8-c9a1-462a-a3de-7acb29f84f86","Type":"ContainerDied","Data":"d03908df57bc643dbe646b1713afa8cff2075b3a3d5fd362429e654e9fb396aa"} Jul 27 21:59:26.252283 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.252259 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:26.374982 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.374944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwwl6\" (UniqueName: \"kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6\") pod \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " Jul 27 21:59:26.375176 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.375021 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util\") pod \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " Jul 27 21:59:26.375176 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.375122 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle\") pod \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\" (UID: \"8df8eef8-c9a1-462a-a3de-7acb29f84f86\") " Jul 27 21:59:26.375504 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.375474 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle" (OuterVolumeSpecName: "bundle") pod "8df8eef8-c9a1-462a-a3de-7acb29f84f86" (UID: "8df8eef8-c9a1-462a-a3de-7acb29f84f86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:26.377304 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.377282 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6" (OuterVolumeSpecName: "kube-api-access-hwwl6") pod "8df8eef8-c9a1-462a-a3de-7acb29f84f86" (UID: "8df8eef8-c9a1-462a-a3de-7acb29f84f86"). InnerVolumeSpecName "kube-api-access-hwwl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:59:26.381756 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.381717 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util" (OuterVolumeSpecName: "util") pod "8df8eef8-c9a1-462a-a3de-7acb29f84f86" (UID: "8df8eef8-c9a1-462a-a3de-7acb29f84f86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:26.476485 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.476405 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:26.476485 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.476430 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df8eef8-c9a1-462a-a3de-7acb29f84f86-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:26.476485 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:26.476439 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwl6\" (UniqueName: \"kubernetes.io/projected/8df8eef8-c9a1-462a-a3de-7acb29f84f86-kube-api-access-hwwl6\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:27.137574 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:27.137540 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" Jul 27 21:59:27.137756 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:27.137538 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f459vw" event={"ID":"8df8eef8-c9a1-462a-a3de-7acb29f84f86","Type":"ContainerDied","Data":"c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318"} Jul 27 21:59:27.137756 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:27.137663 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c126126c709bc9d54717355c48cf2a2c6ec0c149985d73cc23b2645227d49318" Jul 27 21:59:28.865342 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:28.865308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:31.353239 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:31.353201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:31.469527 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:31.469490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:36.863166 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:36.863130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:36.983179 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:36.983124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:38.519770 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.519734 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c"] Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.519964 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="pull" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.519974 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="pull" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.519990 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="util" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.519996 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="util" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.520003 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="extract" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.520009 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="extract" Jul 27 21:59:38.520199 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.520059 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df8eef8-c9a1-462a-a3de-7acb29f84f86" containerName="extract" Jul 27 21:59:38.571898 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.571852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c"] Jul 27 21:59:38.572054 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.571952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.574435 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.574404 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:59:38.575335 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.575320 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:59:38.575463 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.575349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-x9f8r\"" Jul 27 21:59:38.759987 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.759946 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.760141 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.760002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b922j\" (UniqueName: \"kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.760141 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.760082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.774754 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.774690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:38.861397 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.861358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.861569 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.861413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.861569 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.861444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b922j\" (UniqueName: \"kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.861760 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.861739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.861803 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.861773 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.869649 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.869625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b922j\" (UniqueName: \"kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:38.881606 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:38.881585 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:39.027780 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.027753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c"] Jul 27 21:59:39.030313 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:59:39.030279 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617072f5_6387_40d7_b2dd_707ebe4ad51c.slice/crio-d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2 WatchSource:0}: Error finding container d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2: Status 404 returned error can't find the container with id d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2 Jul 27 21:59:39.177879 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.177842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerStarted","Data":"48a9a6ecc2485874f8b902f773ef91e0e8355ca71999fb6b3d94ae50401f958f"} Jul 27 21:59:39.178026 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.177916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerStarted","Data":"d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2"} Jul 27 21:59:39.619571 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.619539 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:39.930326 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.930303 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 21:59:39.931380 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.931356 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 21:59:39.940204 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:39.940143 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 21:59:40.182678 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:40.182581 2559 generic.go:356] "Generic (PLEG): container finished" podID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerID="48a9a6ecc2485874f8b902f773ef91e0e8355ca71999fb6b3d94ae50401f958f" exitCode=0 Jul 27 21:59:40.182678 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:40.182641 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerDied","Data":"48a9a6ecc2485874f8b902f773ef91e0e8355ca71999fb6b3d94ae50401f958f"} Jul 27 21:59:41.187834 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:41.187800 2559 generic.go:356] "Generic (PLEG): container finished" podID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerID="1d8fdf3f64cb52d608a1fffe5371361b40e3354df2653c44f6189d6d02ca4175" exitCode=0 Jul 27 21:59:41.188242 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:41.187907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerDied","Data":"1d8fdf3f64cb52d608a1fffe5371361b40e3354df2653c44f6189d6d02ca4175"} Jul 27 21:59:42.193167 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:42.193116 2559 generic.go:356] "Generic (PLEG): container finished" podID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerID="7944613545d77978544ecf17c5d87a811d9a580e3927202c49deb1184c46f92d" exitCode=0 Jul 27 21:59:42.193540 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:42.193176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerDied","Data":"7944613545d77978544ecf17c5d87a811d9a580e3927202c49deb1184c46f92d"} Jul 27 21:59:43.315040 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.315013 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:43.497302 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.497209 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util\") pod \"617072f5-6387-40d7-b2dd-707ebe4ad51c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " Jul 27 21:59:43.497302 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.497293 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b922j\" (UniqueName: \"kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j\") pod \"617072f5-6387-40d7-b2dd-707ebe4ad51c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " Jul 27 21:59:43.497517 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.497329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle\") pod \"617072f5-6387-40d7-b2dd-707ebe4ad51c\" (UID: \"617072f5-6387-40d7-b2dd-707ebe4ad51c\") " Jul 27 21:59:43.498018 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.497992 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle" (OuterVolumeSpecName: "bundle") pod "617072f5-6387-40d7-b2dd-707ebe4ad51c" (UID: "617072f5-6387-40d7-b2dd-707ebe4ad51c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:43.499484 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.499459 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j" (OuterVolumeSpecName: "kube-api-access-b922j") pod "617072f5-6387-40d7-b2dd-707ebe4ad51c" (UID: "617072f5-6387-40d7-b2dd-707ebe4ad51c"). InnerVolumeSpecName "kube-api-access-b922j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:59:43.502779 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.502761 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util" (OuterVolumeSpecName: "util") pod "617072f5-6387-40d7-b2dd-707ebe4ad51c" (UID: "617072f5-6387-40d7-b2dd-707ebe4ad51c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:59:43.598806 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.598767 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b922j\" (UniqueName: \"kubernetes.io/projected/617072f5-6387-40d7-b2dd-707ebe4ad51c-kube-api-access-b922j\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:43.598806 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.598798 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:43.598806 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:43.598808 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617072f5-6387-40d7-b2dd-707ebe4ad51c-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 21:59:44.202808 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:44.202778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" event={"ID":"617072f5-6387-40d7-b2dd-707ebe4ad51c","Type":"ContainerDied","Data":"d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2"} Jul 27 21:59:44.202808 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:44.202808 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw274c" Jul 27 21:59:44.203010 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:44.202823 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d617ebf5847b8fcba85b4df119fbd2f2881e1c59ff1eab655a2eaf55e4f27bd2" Jul 27 21:59:54.446369 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:54.446331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:54.543672 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:54.543635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:54.841967 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:54.841886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.065528 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.065497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.167246 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.167205 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.241942 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.241836 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8"] Jul 27 21:59:55.242197 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242182 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="pull" Jul 27 21:59:55.242301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242200 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="pull" Jul 27 21:59:55.242301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242211 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="extract" Jul 27 21:59:55.242301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242219 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="extract" Jul 27 21:59:55.242301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242241 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="util" Jul 27 21:59:55.242301 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242248 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="util" Jul 27 21:59:55.242530 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.242311 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="617072f5-6387-40d7-b2dd-707ebe4ad51c" containerName="extract" Jul 27 21:59:55.247940 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.247921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.251099 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.251080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 21:59:55.251233 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.251114 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 21:59:55.251233 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.251186 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pt8cs\"" Jul 27 21:59:55.251658 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.251640 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 21:59:55.251727 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.251687 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 21:59:55.277845 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.277818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8"] Jul 27 21:59:55.284302 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.284279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxxh\" (UniqueName: \"kubernetes.io/projected/b42a6d2f-7d11-4d07-8632-99f8c32c406e-kube-api-access-lnxxh\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.284417 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.284308 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.284417 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.284345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-tmp\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.284417 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.284400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.284540 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.284476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-manifests\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.385553 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.385516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.385886 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.385862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-manifests\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.386249 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.386227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxxh\" (UniqueName: \"kubernetes.io/projected/b42a6d2f-7d11-4d07-8632-99f8c32c406e-kube-api-access-lnxxh\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.386620 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.386604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.386783 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.386770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-tmp\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.387169 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.386541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-manifests\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.389095 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.389066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.389235 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.389219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b42a6d2f-7d11-4d07-8632-99f8c32c406e-tmp\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.389282 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.389268 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b42a6d2f-7d11-4d07-8632-99f8c32c406e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.397189 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.397144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxxh\" (UniqueName: \"kubernetes.io/projected/b42a6d2f-7d11-4d07-8632-99f8c32c406e-kube-api-access-lnxxh\") pod \"opendatahub-operator-controller-manager-5f8fb7b654-q98t8\" (UID: \"b42a6d2f-7d11-4d07-8632-99f8c32c406e\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.458765 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.458670 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.557989 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.557955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 21:59:55.559123 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.559099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.661681 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.661639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.707926 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.707899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8"] Jul 27 21:59:55.710435 ip-10-0-134-139 kubenswrapper[2559]: W0727 21:59:55.710371 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42a6d2f_7d11_4d07_8632_99f8c32c406e.slice/crio-d8362317129add6f94b9e6b41b433095751bd3719cf2dc2190bb7b2cc0b643e8 WatchSource:0}: Error finding container d8362317129add6f94b9e6b41b433095751bd3719cf2dc2190bb7b2cc0b643e8: Status 404 returned error can't find the container with id d8362317129add6f94b9e6b41b433095751bd3719cf2dc2190bb7b2cc0b643e8 Jul 27 21:59:55.712088 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.712073 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:59:55.737344 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.737312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:55.812100 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:55.812060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:56.246305 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:56.246274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" event={"ID":"b42a6d2f-7d11-4d07-8632-99f8c32c406e","Type":"ContainerStarted","Data":"d8362317129add6f94b9e6b41b433095751bd3719cf2dc2190bb7b2cc0b643e8"} Jul 27 21:59:59.260213 ip-10-0-134-139 kubenswrapper[2559]: I0727 21:59:59.260177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" event={"ID":"b42a6d2f-7d11-4d07-8632-99f8c32c406e","Type":"ContainerStarted","Data":"9d2194fb7054df69ac85ab8f7b8533dcc9b5f04273bab21d7734f80203e253a0"} Jul 27 22:00:00.265741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:00.265654 2559 generic.go:356] "Generic (PLEG): container finished" podID="b42a6d2f-7d11-4d07-8632-99f8c32c406e" containerID="9d2194fb7054df69ac85ab8f7b8533dcc9b5f04273bab21d7734f80203e253a0" exitCode=0 Jul 27 22:00:00.265741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:00.265716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" event={"ID":"b42a6d2f-7d11-4d07-8632-99f8c32c406e","Type":"ContainerDied","Data":"9d2194fb7054df69ac85ab8f7b8533dcc9b5f04273bab21d7734f80203e253a0"} Jul 27 22:00:01.271691 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:01.271653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" event={"ID":"b42a6d2f-7d11-4d07-8632-99f8c32c406e","Type":"ContainerStarted","Data":"d20630c3545aa87317b2f143c8192a8572c24735d8d9b63ba7673fc6f02dacda"} Jul 27 22:00:01.272106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:01.271759 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 22:00:01.295375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:01.295321 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" podStartSLOduration=1.873901944 podStartE2EDuration="6.295302559s" totalImagesPullingTime="3.371055641s" totalInitContainerRuntime="1.050344974s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:59:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:01.293815326 +0000 UTC m=+321.674887264" watchObservedRunningTime="2026-07-27 22:00:01.295302559 +0000 UTC m=+321.676374483" Jul 27 22:00:07.366185 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:07.366134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:09.030485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:09.030452 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:12.278535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:12.278502 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fb7b654-q98t8" Jul 27 22:00:14.449809 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:14.449771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:14.612245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:14.612205 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.042903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.042868 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc"] Jul 27 22:00:15.046201 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.046169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.051262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.051237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.055054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.055039 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 22:00:15.055319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.055295 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 22:00:15.068079 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.068054 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc"] Jul 27 22:00:15.137512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.137479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.137512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.137513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tmp\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.137724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.137544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm57\" (UniqueName: \"kubernetes.io/projected/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-kube-api-access-9jm57\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.238243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.238202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.238243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.238248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tmp\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.238482 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:00:15.238363 2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 27 22:00:15.238482 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.238381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm57\" (UniqueName: \"kubernetes.io/projected/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-kube-api-access-9jm57\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.238482 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:00:15.238447 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs podName:8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f nodeName:}" failed. No retries permitted until 2026-07-27 22:00:15.738425649 +0000 UTC m=+336.119497552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs") pod "kube-auth-proxy-5fcc8648dc-l6rzc" (UID: "8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f") : secret "kube-auth-proxy-tls" not found Jul 27 22:00:15.240652 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.240628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tmp\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.243875 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.243845 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5ln2z"] Jul 27 22:00:15.247737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.247718 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.252619 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.252593 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 22:00:15.252836 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.252794 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bv2p6\"" Jul 27 22:00:15.258405 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.258381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5ln2z"] Jul 27 22:00:15.267071 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.267043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm57\" (UniqueName: \"kubernetes.io/projected/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-kube-api-access-9jm57\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.301472 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.301380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.339728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.339693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgbw\" (UniqueName: \"kubernetes.io/projected/25cb3262-3934-42da-add0-9d8d0ba9611e-kube-api-access-fcgbw\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.339901 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.339794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.401614 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.401579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.440859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.440825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgbw\" (UniqueName: \"kubernetes.io/projected/25cb3262-3934-42da-add0-9d8d0ba9611e-kube-api-access-fcgbw\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.441054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.440890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.441054 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:00:15.441002 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 22:00:15.441185 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:00:15.441083 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert podName:25cb3262-3934-42da-add0-9d8d0ba9611e nodeName:}" failed. No retries permitted until 2026-07-27 22:00:15.9410616 +0000 UTC m=+336.322133515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert") pod "odh-model-controller-7998c67dc4-5ln2z" (UID: "25cb3262-3934-42da-add0-9d8d0ba9611e") : secret "odh-model-controller-webhook-cert" not found Jul 27 22:00:15.451690 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.451664 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgbw\" (UniqueName: \"kubernetes.io/projected/25cb3262-3934-42da-add0-9d8d0ba9611e-kube-api-access-fcgbw\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.502555 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.502520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.601779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.601683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.743572 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.743530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.745996 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.745979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f-tls-certs\") pod \"kube-auth-proxy-5fcc8648dc-l6rzc\" (UID: \"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f\") " pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.853892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.853811 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-548j6"] Jul 27 22:00:15.857159 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.857132 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:15.862041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.862004 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.863081 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.863063 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 22:00:15.868993 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.868973 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-548j6"] Jul 27 22:00:15.945571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.945531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.945571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.945571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4rc\" (UniqueName: \"kubernetes.io/projected/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-kube-api-access-tg4rc\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:15.945831 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.945625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-tls-certs\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:15.948129 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.948106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb3262-3934-42da-add0-9d8d0ba9611e-cert\") pod \"odh-model-controller-7998c67dc4-5ln2z\" (UID: \"25cb3262-3934-42da-add0-9d8d0ba9611e\") " pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:15.961098 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.961070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" Jul 27 22:00:15.978030 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:15.978001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:16.046756 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.046728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-tls-certs\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:16.046900 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.046769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4rc\" (UniqueName: \"kubernetes.io/projected/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-kube-api-access-tg4rc\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:16.049485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.049457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-tls-certs\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:16.056312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.056282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4rc\" (UniqueName: \"kubernetes.io/projected/9e4e80ad-ab46-421d-a00e-10ff6cca7e05-kube-api-access-tg4rc\") pod \"model-serving-api-54d89bd5fc-548j6\" (UID: \"9e4e80ad-ab46-421d-a00e-10ff6cca7e05\") " pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:16.099237 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.099207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc"] Jul 27 22:00:16.101053 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:00:16.101029 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce754b3_cb82_48f5_9d5a_2e6e53a92d1f.slice/crio-80fd0039b7037fc73ac54b63a2b32c08cf481e1024455e36f12c6c0e3e4d6e1b WatchSource:0}: Error finding container 80fd0039b7037fc73ac54b63a2b32c08cf481e1024455e36f12c6c0e3e4d6e1b: Status 404 returned error can't find the container with id 80fd0039b7037fc73ac54b63a2b32c08cf481e1024455e36f12c6c0e3e4d6e1b Jul 27 22:00:16.162591 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.162554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:16.167928 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.167907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:16.311582 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.311474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5ln2z"] Jul 27 22:00:16.314650 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:00:16.314620 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cb3262_3934_42da_add0_9d8d0ba9611e.slice/crio-52f2cd90d74c312d89b52fab830bf13213c2a3c4ed6725e3ccbaf3a6a811f4d0 WatchSource:0}: Error finding container 52f2cd90d74c312d89b52fab830bf13213c2a3c4ed6725e3ccbaf3a6a811f4d0: Status 404 returned error can't find the container with id 52f2cd90d74c312d89b52fab830bf13213c2a3c4ed6725e3ccbaf3a6a811f4d0 Jul 27 22:00:16.337189 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.337140 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-548j6"] Jul 27 22:00:16.337288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.337267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" event={"ID":"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f","Type":"ContainerStarted","Data":"80fd0039b7037fc73ac54b63a2b32c08cf481e1024455e36f12c6c0e3e4d6e1b"} Jul 27 22:00:16.338548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.338525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" event={"ID":"25cb3262-3934-42da-add0-9d8d0ba9611e","Type":"ContainerStarted","Data":"52f2cd90d74c312d89b52fab830bf13213c2a3c4ed6725e3ccbaf3a6a811f4d0"} Jul 27 22:00:16.339996 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:00:16.339972 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4e80ad_ab46_421d_a00e_10ff6cca7e05.slice/crio-0d9f2d3c777d7306c4241779ae918df411898ce6717f466c4773999c074a9788 WatchSource:0}: Error finding container 0d9f2d3c777d7306c4241779ae918df411898ce6717f466c4773999c074a9788: Status 404 returned error can't find the container with id 0d9f2d3c777d7306c4241779ae918df411898ce6717f466c4773999c074a9788 Jul 27 22:00:16.485311 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.485233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:16.563187 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:16.563110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:17.345847 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:17.345806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" event={"ID":"9e4e80ad-ab46-421d-a00e-10ff6cca7e05","Type":"ContainerStarted","Data":"0d9f2d3c777d7306c4241779ae918df411898ce6717f466c4773999c074a9788"} Jul 27 22:00:17.551776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:17.551726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:20.377945 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:20.377908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:20.476910 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:20.476882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:21.028021 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.027992 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b"] Jul 27 22:00:21.031341 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.031321 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.033613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.033588 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 22:00:21.033729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.033630 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wv2vd\"" Jul 27 22:00:21.042969 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.042947 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b"] Jul 27 22:00:21.058186 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.058144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:21.088276 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088276 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088311 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzlg\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-kube-api-access-nfzlg\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.088522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.088494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189138 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzlg\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-kube-api-access-nfzlg\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.189789 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.189648 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.190131 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.190065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.190347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.190297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.190564 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.190547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.190838 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.190816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.191861 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.191840 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.192510 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.192492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.200938 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.200913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.202618 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.202589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzlg\" (UniqueName: \"kubernetes.io/projected/54bb52a8-a014-4832-a044-ebc8d6b2ee85-kube-api-access-nfzlg\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crxd6b\" (UID: \"54bb52a8-a014-4832-a044-ebc8d6b2ee85\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.238301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.238267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:21.342941 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.342855 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:21.366560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.366528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" event={"ID":"25cb3262-3934-42da-add0-9d8d0ba9611e","Type":"ContainerStarted","Data":"73e891e56b4889c1958eff33f326c9ad0b6d7457bb7b1598ceed5c7a14a91ce9"} Jul 27 22:00:21.366692 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.366601 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:21.366810 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.366785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:21.368298 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.368272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" event={"ID":"8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f","Type":"ContainerStarted","Data":"c454684447a976d73783b582cbcadeb48be4256ca675625c78d047fb750c011e"} Jul 27 22:00:21.369955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.369934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" event={"ID":"9e4e80ad-ab46-421d-a00e-10ff6cca7e05","Type":"ContainerStarted","Data":"e2b99c7d8b0e55d77d0dafd63b7574410e81ea57cf915ae56b048bdc2fd6d58e"} Jul 27 22:00:21.370051 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.369992 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:21.385265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.385206 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" podStartSLOduration=2.285301153 podStartE2EDuration="6.385184197s" totalImagesPullingTime="4.099883044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:21.384107576 +0000 UTC m=+341.765179501" watchObservedRunningTime="2026-07-27 22:00:21.385184197 +0000 UTC m=+341.766256120" Jul 27 22:00:21.400948 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.400902 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fcc8648dc-l6rzc" podStartSLOduration=2.095282209 podStartE2EDuration="6.400881252s" totalImagesPullingTime="4.305599043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:21.400780287 +0000 UTC m=+341.781852213" watchObservedRunningTime="2026-07-27 22:00:21.400881252 +0000 UTC m=+341.781953176" Jul 27 22:00:21.421660 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.421550 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" podStartSLOduration=2.358313118 podStartE2EDuration="6.421532392s" totalImagesPullingTime="4.063219274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:21.42021765 +0000 UTC m=+341.801289577" watchObservedRunningTime="2026-07-27 22:00:21.421532392 +0000 UTC m=+341.802604315" Jul 27 22:00:21.494383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:21.494258 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b"] Jul 27 22:00:21.496817 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:00:21.496790 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bb52a8_a014_4832_a044_ebc8d6b2ee85.slice/crio-869e4ba25270834ab9cbcfb45d759a9a2d846d6df47ce3ebb050abc2a55a742e WatchSource:0}: Error finding container 869e4ba25270834ab9cbcfb45d759a9a2d846d6df47ce3ebb050abc2a55a742e: Status 404 returned error can't find the container with id 869e4ba25270834ab9cbcfb45d759a9a2d846d6df47ce3ebb050abc2a55a742e Jul 27 22:00:22.375226 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:22.375195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" event={"ID":"54bb52a8-a014-4832-a044-ebc8d6b2ee85","Type":"ContainerStarted","Data":"869e4ba25270834ab9cbcfb45d759a9a2d846d6df47ce3ebb050abc2a55a742e"} Jul 27 22:00:23.930095 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.930058 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5"] Jul 27 22:00:23.937237 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.937213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:23.940096 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.940071 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 22:00:23.940096 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.940088 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 22:00:23.941366 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.941342 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 27 22:00:23.941464 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.941345 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 27 22:00:23.941464 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.941379 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 27 22:00:23.941558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.941484 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-58jf7\"" Jul 27 22:00:23.945284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:23.945260 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5"] Jul 27 22:00:24.015697 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.015660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.015865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.015741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-metrics-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.015865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.015774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d39fa14-2cf3-4fb3-9474-16c4992b379c-manager-config\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.015865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.015834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd9h\" (UniqueName: \"kubernetes.io/projected/6d39fa14-2cf3-4fb3-9474-16c4992b379c-kube-api-access-rrd9h\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.116792 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.116757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.117002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.116831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-metrics-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.117002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.116872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d39fa14-2cf3-4fb3-9474-16c4992b379c-manager-config\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.117002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.116918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd9h\" (UniqueName: \"kubernetes.io/projected/6d39fa14-2cf3-4fb3-9474-16c4992b379c-kube-api-access-rrd9h\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.118058 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.118034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d39fa14-2cf3-4fb3-9474-16c4992b379c-manager-config\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.120090 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.120066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.121096 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.121071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d39fa14-2cf3-4fb3-9474-16c4992b379c-metrics-cert\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.148516 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.148489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd9h\" (UniqueName: \"kubernetes.io/projected/6d39fa14-2cf3-4fb3-9474-16c4992b379c-kube-api-access-rrd9h\") pod \"lws-controller-manager-8444c4f6f9-9zrk5\" (UID: \"6d39fa14-2cf3-4fb3-9474-16c4992b379c\") " pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.248832 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.248739 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:24.806616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.806568 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:24.806714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.806670 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:24.806714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.806708 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:24.873349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:24.873318 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5"] Jul 27 22:00:24.876286 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:00:24.876254 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d39fa14_2cf3_4fb3_9474_16c4992b379c.slice/crio-aadbccff7206c5fc1ad06a4dd53e993d696c742533193542b806f3b3c63cadd7 WatchSource:0}: Error finding container aadbccff7206c5fc1ad06a4dd53e993d696c742533193542b806f3b3c63cadd7: Status 404 returned error can't find the container with id aadbccff7206c5fc1ad06a4dd53e993d696c742533193542b806f3b3c63cadd7 Jul 27 22:00:25.390594 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:25.390563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" event={"ID":"54bb52a8-a014-4832-a044-ebc8d6b2ee85","Type":"ContainerStarted","Data":"e696c41bfb19e00283e30b204e0250e1ca4eba434bf08ec9c8c7086c84d64d2d"} Jul 27 22:00:25.391726 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:25.391696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" event={"ID":"6d39fa14-2cf3-4fb3-9474-16c4992b379c","Type":"ContainerStarted","Data":"aadbccff7206c5fc1ad06a4dd53e993d696c742533193542b806f3b3c63cadd7"} Jul 27 22:00:25.411076 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:25.411030 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" podStartSLOduration=1.104071979 podStartE2EDuration="4.411015535s" totalImagesPullingTime="3.306943556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:25.409527571 +0000 UTC m=+345.790599495" watchObservedRunningTime="2026-07-27 22:00:25.411015535 +0000 UTC m=+345.792087459" Jul 27 22:00:26.343860 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.343830 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:26.349356 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.349324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:26.397889 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.397854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:26.398834 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.398813 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crxd6b" Jul 27 22:00:26.732308 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.732271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:26.838793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:26.838762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.135794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:27.135758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.221548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:27.221503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.403590 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:27.403490 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" event={"ID":"6d39fa14-2cf3-4fb3-9474-16c4992b379c","Type":"ContainerStarted","Data":"ac56f3d583f7698540c9ef488a992a9e79e3c79970ce5f00e489d27407756497"} Jul 27 22:00:27.403972 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:27.403630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:00:27.426583 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:27.424319 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" podStartSLOduration=2.788475568 podStartE2EDuration="4.424298738s" totalImagesPullingTime="1.63582317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:27.421022624 +0000 UTC m=+347.802094548" watchObservedRunningTime="2026-07-27 22:00:27.424298738 +0000 UTC m=+347.805370662" Jul 27 22:00:28.002881 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:28.002842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:28.103543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:28.103491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:28.170737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:28.170701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:32.378267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:32.378183 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-5ln2z" Jul 27 22:00:32.381955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:32.381923 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-548j6" Jul 27 22:00:38.410281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:00:38.410246 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8444c4f6f9-9zrk5" Jul 27 22:01:04.755779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:04.755734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:14.862523 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:14.862487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:14.919809 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:14.919782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:15.908846 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:15.908793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:16.017567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:16.017532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:16.102362 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:16.102323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:16.721660 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:16.721621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:16.819725 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:16.819687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:31.379057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.379022 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp"] Jul 27 22:01:31.381272 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.381255 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.383405 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.383380 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 22:01:31.383522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.383422 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4h78d\"" Jul 27 22:01:31.384190 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.384173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 22:01:31.391343 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.391321 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp"] Jul 27 22:01:31.430561 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.430529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lb2\" (UniqueName: \"kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.430745 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.430568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.430745 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.430626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.531651 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.531592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lb2\" (UniqueName: \"kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.531798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.531674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.531798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.531716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.532097 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.532072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.532195 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.532098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.541499 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.541477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lb2\" (UniqueName: \"kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.691066 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.690975 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:31.823942 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.823914 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp"] Jul 27 22:01:31.827425 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:01:31.827400 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f06467_3337_483e_88c0_6f3513060a38.slice/crio-e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64 WatchSource:0}: Error finding container e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64: Status 404 returned error can't find the container with id e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64 Jul 27 22:01:31.977547 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.977505 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v"] Jul 27 22:01:31.979929 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.979907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:31.990492 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:31.990471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v"] Jul 27 22:01:32.036462 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.036435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pz5f\" (UniqueName: \"kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.036613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.036496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.036613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.036539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.136940 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.136902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.137120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.136962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pz5f\" (UniqueName: \"kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.137120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.137055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.137390 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.137368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.137485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.137401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.145515 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.145495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pz5f\" (UniqueName: \"kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.317620 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.317540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:32.446448 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.446402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v"] Jul 27 22:01:32.448883 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:01:32.448841 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6363ef5_506f_4f64_a544_fd95ac4653e2.slice/crio-4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564 WatchSource:0}: Error finding container 4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564: Status 404 returned error can't find the container with id 4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564 Jul 27 22:01:32.586022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.585961 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf"] Jul 27 22:01:32.588226 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.588209 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.598829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.598805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf"] Jul 27 22:01:32.641915 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.641877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6z4\" (UniqueName: \"kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.642062 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.641930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.642062 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.642006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.669318 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.669289 2559 generic.go:356] "Generic (PLEG): container finished" podID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerID="0de8cbbef4818b5ee571eadab1dac27386ea16601e7b16a1edf298e4db830ba8" exitCode=0 Jul 27 22:01:32.669479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.669374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" event={"ID":"e6363ef5-506f-4f64-a544-fd95ac4653e2","Type":"ContainerDied","Data":"0de8cbbef4818b5ee571eadab1dac27386ea16601e7b16a1edf298e4db830ba8"} Jul 27 22:01:32.669479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.669427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" event={"ID":"e6363ef5-506f-4f64-a544-fd95ac4653e2","Type":"ContainerStarted","Data":"4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564"} Jul 27 22:01:32.670860 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.670835 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5f06467-3337-483e-88c0-6f3513060a38" containerID="e41a66f3c59fa53b6c0b546081f131f5c4622c731f87e3981bb528e1f2fcbbc5" exitCode=0 Jul 27 22:01:32.670951 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.670876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" event={"ID":"f5f06467-3337-483e-88c0-6f3513060a38","Type":"ContainerDied","Data":"e41a66f3c59fa53b6c0b546081f131f5c4622c731f87e3981bb528e1f2fcbbc5"} Jul 27 22:01:32.670951 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.670905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" event={"ID":"f5f06467-3337-483e-88c0-6f3513060a38","Type":"ContainerStarted","Data":"e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64"} Jul 27 22:01:32.742365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.742335 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6z4\" (UniqueName: \"kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.742365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.742368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.742555 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.742406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.742782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.742761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.742822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.742793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.750320 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.750300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6z4\" (UniqueName: \"kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.918048 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.918018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:32.976787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.976762 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws"] Jul 27 22:01:32.979363 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.979344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:32.988548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:32.988525 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws"] Jul 27 22:01:33.045119 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.045084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.045289 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.045131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67g78\" (UniqueName: \"kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.045289 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.045176 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.051067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.051005 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf"] Jul 27 22:01:33.053483 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:01:33.053460 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b626c95_aa0b_4da3_930a_df485f8e541d.slice/crio-f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c WatchSource:0}: Error finding container f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c: Status 404 returned error can't find the container with id f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c Jul 27 22:01:33.145598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.145567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67g78\" (UniqueName: \"kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.145721 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.145602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.145721 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.145664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.145962 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.145943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.146037 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.146007 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.153903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.153883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67g78\" (UniqueName: \"kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.290005 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.289923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:33.427477 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.427447 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws"] Jul 27 22:01:33.471772 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:01:33.471742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201d9de_dffb_41a7_b193_a62458ab7f55.slice/crio-1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44 WatchSource:0}: Error finding container 1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44: Status 404 returned error can't find the container with id 1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44 Jul 27 22:01:33.676862 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.676830 2559 generic.go:356] "Generic (PLEG): container finished" podID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerID="3e78ee128321ab45063727ff5a1e719ac892ffaaf1ba5b35be7058694916d1ce" exitCode=0 Jul 27 22:01:33.676995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.676891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" event={"ID":"0201d9de-dffb-41a7-b193-a62458ab7f55","Type":"ContainerDied","Data":"3e78ee128321ab45063727ff5a1e719ac892ffaaf1ba5b35be7058694916d1ce"} Jul 27 22:01:33.676995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.676934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" event={"ID":"0201d9de-dffb-41a7-b193-a62458ab7f55","Type":"ContainerStarted","Data":"1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44"} Jul 27 22:01:33.678970 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.678940 2559 generic.go:356] "Generic (PLEG): container finished" podID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerID="6740b6059d51dbcc8acf6da280d572b5cf8e4f8eeca33edb9d95c37b0195ed5a" exitCode=0 Jul 27 22:01:33.679103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.679014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" event={"ID":"e6363ef5-506f-4f64-a544-fd95ac4653e2","Type":"ContainerDied","Data":"6740b6059d51dbcc8acf6da280d572b5cf8e4f8eeca33edb9d95c37b0195ed5a"} Jul 27 22:01:33.681429 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.681411 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5f06467-3337-483e-88c0-6f3513060a38" containerID="3565d22c24520433f95022c5d9bab094ed182f5de1baab1b53123d17a2d0b69c" exitCode=0 Jul 27 22:01:33.681524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.681476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" event={"ID":"f5f06467-3337-483e-88c0-6f3513060a38","Type":"ContainerDied","Data":"3565d22c24520433f95022c5d9bab094ed182f5de1baab1b53123d17a2d0b69c"} Jul 27 22:01:33.683038 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.682982 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerID="509dabf2ef714b1052ae38a16bbe735a5e63d0c56089cc38d8026c0396678dbe" exitCode=0 Jul 27 22:01:33.683144 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.683043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" event={"ID":"3b626c95-aa0b-4da3-930a-df485f8e541d","Type":"ContainerDied","Data":"509dabf2ef714b1052ae38a16bbe735a5e63d0c56089cc38d8026c0396678dbe"} Jul 27 22:01:33.683144 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:33.683067 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" event={"ID":"3b626c95-aa0b-4da3-930a-df485f8e541d","Type":"ContainerStarted","Data":"f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c"} Jul 27 22:01:34.689571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.689493 2559 generic.go:356] "Generic (PLEG): container finished" podID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerID="2949341d2cd5d14e088f231b573ac17873b203f48291694e471d050bf69287b7" exitCode=0 Jul 27 22:01:34.689925 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.689571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" event={"ID":"e6363ef5-506f-4f64-a544-fd95ac4653e2","Type":"ContainerDied","Data":"2949341d2cd5d14e088f231b573ac17873b203f48291694e471d050bf69287b7"} Jul 27 22:01:34.691489 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.691469 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5f06467-3337-483e-88c0-6f3513060a38" containerID="20f43c924d27ab378638b8bae9d2b4d2633f3fa37418199e396fe405bd502f73" exitCode=0 Jul 27 22:01:34.691640 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.691529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" event={"ID":"f5f06467-3337-483e-88c0-6f3513060a38","Type":"ContainerDied","Data":"20f43c924d27ab378638b8bae9d2b4d2633f3fa37418199e396fe405bd502f73"} Jul 27 22:01:34.693124 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.693103 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerID="b037e9312852cf290f5129a7f6a18e056235e0856042c820f12ccb427f7b96a2" exitCode=0 Jul 27 22:01:34.693247 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.693193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" event={"ID":"3b626c95-aa0b-4da3-930a-df485f8e541d","Type":"ContainerDied","Data":"b037e9312852cf290f5129a7f6a18e056235e0856042c820f12ccb427f7b96a2"} Jul 27 22:01:34.694914 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.694893 2559 generic.go:356] "Generic (PLEG): container finished" podID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerID="ac75d8a320af1eb27dc21e69ad9d9273a6873064d892b4816b8183598e84ad44" exitCode=0 Jul 27 22:01:34.695028 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:34.694954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" event={"ID":"0201d9de-dffb-41a7-b193-a62458ab7f55","Type":"ContainerDied","Data":"ac75d8a320af1eb27dc21e69ad9d9273a6873064d892b4816b8183598e84ad44"} Jul 27 22:01:35.701658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.701619 2559 generic.go:356] "Generic (PLEG): container finished" podID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerID="d2eb1994d12c0ea842b7b191a0d19413911681b1073a7e895acea76cb2666887" exitCode=0 Jul 27 22:01:35.702040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.701712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" event={"ID":"0201d9de-dffb-41a7-b193-a62458ab7f55","Type":"ContainerDied","Data":"d2eb1994d12c0ea842b7b191a0d19413911681b1073a7e895acea76cb2666887"} Jul 27 22:01:35.703549 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.703529 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerID="d61e27abf91cb94dd9c0d03686c602c7c2caf9fc8d6e43a7e2ddae25f1a9efe4" exitCode=0 Jul 27 22:01:35.703691 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.703626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" event={"ID":"3b626c95-aa0b-4da3-930a-df485f8e541d","Type":"ContainerDied","Data":"d61e27abf91cb94dd9c0d03686c602c7c2caf9fc8d6e43a7e2ddae25f1a9efe4"} Jul 27 22:01:35.855111 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.855086 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:35.857798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.857778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:35.966953 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.966854 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle\") pod \"f5f06467-3337-483e-88c0-6f3513060a38\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " Jul 27 22:01:35.967104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.966995 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pz5f\" (UniqueName: \"kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f\") pod \"e6363ef5-506f-4f64-a544-fd95ac4653e2\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " Jul 27 22:01:35.967104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util\") pod \"f5f06467-3337-483e-88c0-6f3513060a38\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " Jul 27 22:01:35.967104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util\") pod \"e6363ef5-506f-4f64-a544-fd95ac4653e2\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " Jul 27 22:01:35.967295 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle\") pod \"e6363ef5-506f-4f64-a544-fd95ac4653e2\" (UID: \"e6363ef5-506f-4f64-a544-fd95ac4653e2\") " Jul 27 22:01:35.967295 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967222 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7lb2\" (UniqueName: \"kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2\") pod \"f5f06467-3337-483e-88c0-6f3513060a38\" (UID: \"f5f06467-3337-483e-88c0-6f3513060a38\") " Jul 27 22:01:35.967690 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967660 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle" (OuterVolumeSpecName: "bundle") pod "f5f06467-3337-483e-88c0-6f3513060a38" (UID: "f5f06467-3337-483e-88c0-6f3513060a38"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:35.967963 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.967747 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle" (OuterVolumeSpecName: "bundle") pod "e6363ef5-506f-4f64-a544-fd95ac4653e2" (UID: "e6363ef5-506f-4f64-a544-fd95ac4653e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:35.969433 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.969410 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f" (OuterVolumeSpecName: "kube-api-access-9pz5f") pod "e6363ef5-506f-4f64-a544-fd95ac4653e2" (UID: "e6363ef5-506f-4f64-a544-fd95ac4653e2"). InnerVolumeSpecName "kube-api-access-9pz5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:35.969823 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.969795 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2" (OuterVolumeSpecName: "kube-api-access-d7lb2") pod "f5f06467-3337-483e-88c0-6f3513060a38" (UID: "f5f06467-3337-483e-88c0-6f3513060a38"). InnerVolumeSpecName "kube-api-access-d7lb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:35.972436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.972416 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util" (OuterVolumeSpecName: "util") pod "f5f06467-3337-483e-88c0-6f3513060a38" (UID: "f5f06467-3337-483e-88c0-6f3513060a38"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:35.973614 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:35.973487 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util" (OuterVolumeSpecName: "util") pod "e6363ef5-506f-4f64-a544-fd95ac4653e2" (UID: "e6363ef5-506f-4f64-a544-fd95ac4653e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:36.068428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068376 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7lb2\" (UniqueName: \"kubernetes.io/projected/f5f06467-3337-483e-88c0-6f3513060a38-kube-api-access-d7lb2\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.068428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068422 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.068428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068435 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pz5f\" (UniqueName: \"kubernetes.io/projected/e6363ef5-506f-4f64-a544-fd95ac4653e2-kube-api-access-9pz5f\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.068663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068448 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5f06467-3337-483e-88c0-6f3513060a38-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.068663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068461 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.068663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.068470 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6363ef5-506f-4f64-a544-fd95ac4653e2-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:36.715538 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.715505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" event={"ID":"e6363ef5-506f-4f64-a544-fd95ac4653e2","Type":"ContainerDied","Data":"4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564"} Jul 27 22:01:36.715995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.715517 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v" Jul 27 22:01:36.715995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.715597 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a965b42ef9f9a36fade6ee12c2b618cbe0a9dce8887d05483b3815083048564" Jul 27 22:01:36.717586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.717552 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" Jul 27 22:01:36.717710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.717594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp" event={"ID":"f5f06467-3337-483e-88c0-6f3513060a38","Type":"ContainerDied","Data":"e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64"} Jul 27 22:01:36.717710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.717643 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e978cddbb8ff7f3946c17e3649a6664348b35aad6bcca3c0cd5807da79aa5b64" Jul 27 22:01:36.858022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.858002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:36.860210 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.860193 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:36.975760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975667 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n6z4\" (UniqueName: \"kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4\") pod \"3b626c95-aa0b-4da3-930a-df485f8e541d\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " Jul 27 22:01:36.975760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util\") pod \"0201d9de-dffb-41a7-b193-a62458ab7f55\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " Jul 27 22:01:36.975976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975785 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67g78\" (UniqueName: \"kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78\") pod \"0201d9de-dffb-41a7-b193-a62458ab7f55\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " Jul 27 22:01:36.975976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975820 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle\") pod \"0201d9de-dffb-41a7-b193-a62458ab7f55\" (UID: \"0201d9de-dffb-41a7-b193-a62458ab7f55\") " Jul 27 22:01:36.975976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util\") pod \"3b626c95-aa0b-4da3-930a-df485f8e541d\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " Jul 27 22:01:36.975976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.975893 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle\") pod \"3b626c95-aa0b-4da3-930a-df485f8e541d\" (UID: \"3b626c95-aa0b-4da3-930a-df485f8e541d\") " Jul 27 22:01:36.976713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.976384 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle" (OuterVolumeSpecName: "bundle") pod "0201d9de-dffb-41a7-b193-a62458ab7f55" (UID: "0201d9de-dffb-41a7-b193-a62458ab7f55"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:36.976713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.976470 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle" (OuterVolumeSpecName: "bundle") pod "3b626c95-aa0b-4da3-930a-df485f8e541d" (UID: "3b626c95-aa0b-4da3-930a-df485f8e541d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:36.978008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.977988 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4" (OuterVolumeSpecName: "kube-api-access-2n6z4") pod "3b626c95-aa0b-4da3-930a-df485f8e541d" (UID: "3b626c95-aa0b-4da3-930a-df485f8e541d"). InnerVolumeSpecName "kube-api-access-2n6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:36.978111 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.978037 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78" (OuterVolumeSpecName: "kube-api-access-67g78") pod "0201d9de-dffb-41a7-b193-a62458ab7f55" (UID: "0201d9de-dffb-41a7-b193-a62458ab7f55"). InnerVolumeSpecName "kube-api-access-67g78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:36.981796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.981764 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util" (OuterVolumeSpecName: "util") pod "0201d9de-dffb-41a7-b193-a62458ab7f55" (UID: "0201d9de-dffb-41a7-b193-a62458ab7f55"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:36.982114 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:36.982093 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util" (OuterVolumeSpecName: "util") pod "3b626c95-aa0b-4da3-930a-df485f8e541d" (UID: "3b626c95-aa0b-4da3-930a-df485f8e541d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:37.076588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076534 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67g78\" (UniqueName: \"kubernetes.io/projected/0201d9de-dffb-41a7-b193-a62458ab7f55-kube-api-access-67g78\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.076588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076586 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.076588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076600 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.076844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076638 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b626c95-aa0b-4da3-930a-df485f8e541d-bundle\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.076844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076650 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6z4\" (UniqueName: \"kubernetes.io/projected/3b626c95-aa0b-4da3-930a-df485f8e541d-kube-api-access-2n6z4\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.076844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.076662 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0201d9de-dffb-41a7-b193-a62458ab7f55-util\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:01:37.723553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.723528 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" Jul 27 22:01:37.723553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.723551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf" event={"ID":"3b626c95-aa0b-4da3-930a-df485f8e541d","Type":"ContainerDied","Data":"f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c"} Jul 27 22:01:37.724029 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.723622 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ba9f67094428b959cb599298ceba72c7d0e202d6e1cbcd023f06d46a839a8c" Jul 27 22:01:37.725398 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.725378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" event={"ID":"0201d9de-dffb-41a7-b193-a62458ab7f55","Type":"ContainerDied","Data":"1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44"} Jul 27 22:01:37.725494 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.725409 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cdae126366be50612d6870bbba1e88af6019d5dc79fbdc809eb0e826438dd44" Jul 27 22:01:37.725494 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:37.725415 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws" Jul 27 22:01:39.048194 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:39.048143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:39.640714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:39.640676 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:40.879835 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:40.879790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:40.984759 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:40.984719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:43.457521 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:43.457486 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:43.771081 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:43.770991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:47.432419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:47.432383 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:47.563423 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:47.563389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:49.484316 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:49.484285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:51.309094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:51.309057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:52.451120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:52.451095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:52.509061 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:52.509023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:52.853906 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:52.853823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:54.121511 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:54.121475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:55.753381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:55.753354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:55.813490 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:55.813464 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:55.874064 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:55.874024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:56.904868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:56.904834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:57.005448 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:57.005415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:57.299460 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:57.299367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:57.834520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:01:57.834481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:04.408560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:04.408476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:04.472923 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:04.472888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:06.639948 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:06.639914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:06.742273 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:06.742238 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:06.840762 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:06.840729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:08.311366 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:08.311336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:08.413365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:08.413332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:12.069426 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:12.069389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:12.130720 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:12.130675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:12.274044 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:12.274014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:12.371094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:12.371067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:14.826575 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:14.826545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:14.927302 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:14.927263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:21.318254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:21.318209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:21.418504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:21.418473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:26.073462 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:26.073425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:26.173108 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:26.173073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:30.078887 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:30.078842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:30.142837 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:30.142804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:30.684314 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:30.684284 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:30.762081 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:30.762050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:34.289948 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:34.289912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:34.395205 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:34.395144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:42.303656 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:42.303621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:42.381597 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:42.381548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:46.275446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:46.275408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:46.377574 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:46.377538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:48.964950 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.964916 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4"] Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965236 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965249 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965264 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965269 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965277 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965282 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965289 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965294 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965301 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965306 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965310 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965315 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965324 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965331 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965337 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965342 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965348 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965353 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="util" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965360 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965365 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965373 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965379 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="pull" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965385 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="extract" Jul 27 22:02:48.965380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965391 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="extract" Jul 27 22:02:48.966135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965440 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0201d9de-dffb-41a7-b193-a62458ab7f55" containerName="extract" Jul 27 22:02:48.966135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965449 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6363ef5-506f-4f64-a544-fd95ac4653e2" containerName="extract" Jul 27 22:02:48.966135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965455 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b626c95-aa0b-4da3-930a-df485f8e541d" containerName="extract" Jul 27 22:02:48.966135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.965459 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f06467-3337-483e-88c0-6f3513060a38" containerName="extract" Jul 27 22:02:48.968308 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.968292 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:48.970569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.970549 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-242rp\"" Jul 27 22:02:48.977309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.977286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4"] Jul 27 22:02:48.994497 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:48.994457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:49.083832 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96892165-bb45-4963-98fd-084a637f2b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.083832 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96892165-bb45-4963-98fd-084a637f2b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083937 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.083985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lgp\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-kube-api-access-m6lgp\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.084068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.084101 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.084193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.084124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185224 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lgp\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-kube-api-access-m6lgp\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185676 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185676 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96892165-bb45-4963-98fd-084a637f2b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185691 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96892165-bb45-4963-98fd-084a637f2b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.185916 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.185787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.186144 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.186125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.186243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.186224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.186291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.186261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96892165-bb45-4963-98fd-084a637f2b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.187717 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.187698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96892165-bb45-4963-98fd-084a637f2b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.188189 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.188174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96892165-bb45-4963-98fd-084a637f2b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.193787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.193762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.196980 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.196955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lgp\" (UniqueName: \"kubernetes.io/projected/96892165-bb45-4963-98fd-084a637f2b7d-kube-api-access-m6lgp\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rg9d4\" (UID: \"96892165-bb45-4963-98fd-084a637f2b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.279503 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.279418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:49.422218 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.422194 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4"] Jul 27 22:02:49.424886 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:02:49.424857 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96892165_bb45_4963_98fd_084a637f2b7d.slice/crio-05f66f3cf04dd3db5f8e14c0b42e2fdc4d28812ef49c19d79a505b8ef3ed7f03 WatchSource:0}: Error finding container 05f66f3cf04dd3db5f8e14c0b42e2fdc4d28812ef49c19d79a505b8ef3ed7f03: Status 404 returned error can't find the container with id 05f66f3cf04dd3db5f8e14c0b42e2fdc4d28812ef49c19d79a505b8ef3ed7f03 Jul 27 22:02:49.427045 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.427013 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:49.427102 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.427083 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:49.427143 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.427112 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:49.536952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:49.536856 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:50.059867 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:50.059832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" event={"ID":"96892165-bb45-4963-98fd-084a637f2b7d","Type":"ContainerStarted","Data":"bac274339fe20123226234650893438c816e719c6e8bd19412f2abc7d98de1fe"} Jul 27 22:02:50.060310 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:50.059882 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" event={"ID":"96892165-bb45-4963-98fd-084a637f2b7d","Type":"ContainerStarted","Data":"05f66f3cf04dd3db5f8e14c0b42e2fdc4d28812ef49c19d79a505b8ef3ed7f03"} Jul 27 22:02:50.083117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:50.083060 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" podStartSLOduration=2.083042699 podStartE2EDuration="2.083042699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:02:50.080187454 +0000 UTC m=+490.461259378" watchObservedRunningTime="2026-07-27 22:02:50.083042699 +0000 UTC m=+490.464114625" Jul 27 22:02:50.279716 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:50.279677 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:50.284743 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:50.284716 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:51.064822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:51.064787 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:51.065862 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:51.065842 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:02:52.313484 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:52.313439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:52.418839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:52.418806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:52.479195 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:52.479136 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:53.163002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.162966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:53.248809 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.248776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:53.272063 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.272023 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:02:53.275500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.275475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:02:53.278242 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.278208 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 22:02:53.278839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.278826 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7r4kf\"" Jul 27 22:02:53.278889 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.278859 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 22:02:53.284753 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.284729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:02:53.425098 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.425008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbrx\" (UniqueName: \"kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx\") pod \"authorino-75d556bd77-lf5lk\" (UID: \"ecb8bac5-8c1c-438d-8a78-1b193a317b5a\") " pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:02:53.526202 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.526141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbrx\" (UniqueName: \"kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx\") pod \"authorino-75d556bd77-lf5lk\" (UID: \"ecb8bac5-8c1c-438d-8a78-1b193a317b5a\") " pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:02:53.545636 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.545603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbrx\" (UniqueName: \"kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx\") pod \"authorino-75d556bd77-lf5lk\" (UID: \"ecb8bac5-8c1c-438d-8a78-1b193a317b5a\") " pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:02:53.586295 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.586247 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:02:53.725907 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:53.725827 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:02:53.729450 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:02:53.729420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb8bac5_8c1c_438d_8a78_1b193a317b5a.slice/crio-be3677bec2d12a05a7d007ab503b39f4e8d30124aa3c7cd1d93076dfd2507a51 WatchSource:0}: Error finding container be3677bec2d12a05a7d007ab503b39f4e8d30124aa3c7cd1d93076dfd2507a51: Status 404 returned error can't find the container with id be3677bec2d12a05a7d007ab503b39f4e8d30124aa3c7cd1d93076dfd2507a51 Jul 27 22:02:54.083735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:54.083652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lf5lk" event={"ID":"ecb8bac5-8c1c-438d-8a78-1b193a317b5a","Type":"ContainerStarted","Data":"be3677bec2d12a05a7d007ab503b39f4e8d30124aa3c7cd1d93076dfd2507a51"} Jul 27 22:02:57.101359 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:57.101314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lf5lk" event={"ID":"ecb8bac5-8c1c-438d-8a78-1b193a317b5a","Type":"ContainerStarted","Data":"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237"} Jul 27 22:02:57.116592 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:02:57.116540 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lf5lk" podStartSLOduration=1.264874801 podStartE2EDuration="4.116521977s" totalImagesPullingTime="2.851647176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:02:57.115360714 +0000 UTC m=+497.496432637" watchObservedRunningTime="2026-07-27 22:02:57.116521977 +0000 UTC m=+497.497593901" Jul 27 22:03:12.349053 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:12.349017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:12.449667 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:12.449628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:12.867575 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:12.867534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:25.598911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.598874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:25.698869 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.698829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:25.815299 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.815245 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:25.819504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.819482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:25.822965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.822931 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:25.883922 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.883885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2tb\" (UniqueName: \"kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb\") pod \"authorino-5d4b67c446-bqltw\" (UID: \"52efcc9e-7e46-435e-9b95-30ae552db588\") " pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:25.985255 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.985211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2tb\" (UniqueName: \"kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb\") pod \"authorino-5d4b67c446-bqltw\" (UID: \"52efcc9e-7e46-435e-9b95-30ae552db588\") " pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:25.993802 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:25.993770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2tb\" (UniqueName: \"kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb\") pod \"authorino-5d4b67c446-bqltw\" (UID: \"52efcc9e-7e46-435e-9b95-30ae552db588\") " pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:26.052919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.052881 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:26.053175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.053144 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:26.073219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.073187 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cc74f569d-b9cmd"] Jul 27 22:03:26.075424 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.075401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.085817 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.085791 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cc74f569d-b9cmd"] Jul 27 22:03:26.120773 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.120492 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cc74f569d-b9cmd"] Jul 27 22:03:26.120773 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:03:26.120736 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hjhrs], unattached volumes=[], failed to process volumes=[kube-api-access-hjhrs]: context canceled" pod="kuadrant-system/authorino-5cc74f569d-b9cmd" podUID="6a5c7fee-3cad-48ab-be8c-b4777e953982" Jul 27 22:03:26.148652 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.148617 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:03:26.150733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.150711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.153259 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.153235 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 22:03:26.159786 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.159758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:03:26.186822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.186794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.186974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.186888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhrs\" (UniqueName: \"kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs\") pod \"authorino-5cc74f569d-b9cmd\" (UID: \"6a5c7fee-3cad-48ab-be8c-b4777e953982\") " pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.186974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.186934 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbgb\" (UniqueName: \"kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.201988 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.201963 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:26.204543 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:03:26.204517 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52efcc9e_7e46_435e_9b95_30ae552db588.slice/crio-7d81ca75dc5761a7efe1c607729022294982b16ce9a2a99f04d566e2f8174be3 WatchSource:0}: Error finding container 7d81ca75dc5761a7efe1c607729022294982b16ce9a2a99f04d566e2f8174be3: Status 404 returned error can't find the container with id 7d81ca75dc5761a7efe1c607729022294982b16ce9a2a99f04d566e2f8174be3 Jul 27 22:03:26.235206 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.235179 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.235206 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.235182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqltw" event={"ID":"52efcc9e-7e46-435e-9b95-30ae552db588","Type":"ContainerStarted","Data":"7d81ca75dc5761a7efe1c607729022294982b16ce9a2a99f04d566e2f8174be3"} Jul 27 22:03:26.241559 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.241534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.287651 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.287620 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhrs\" (UniqueName: \"kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs\") pod \"authorino-5cc74f569d-b9cmd\" (UID: \"6a5c7fee-3cad-48ab-be8c-b4777e953982\") " pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.287859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.287658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbgb\" (UniqueName: \"kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.287859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.287702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.290380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.290351 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.296256 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.296229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhrs\" (UniqueName: \"kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs\") pod \"authorino-5cc74f569d-b9cmd\" (UID: \"6a5c7fee-3cad-48ab-be8c-b4777e953982\") " pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:26.296461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.296229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbgb\" (UniqueName: \"kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb\") pod \"authorino-6f766b6554-lp7xx\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.388579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.388532 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjhrs\" (UniqueName: \"kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs\") pod \"6a5c7fee-3cad-48ab-be8c-b4777e953982\" (UID: \"6a5c7fee-3cad-48ab-be8c-b4777e953982\") " Jul 27 22:03:26.390821 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.390796 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs" (OuterVolumeSpecName: "kube-api-access-hjhrs") pod "6a5c7fee-3cad-48ab-be8c-b4777e953982" (UID: "6a5c7fee-3cad-48ab-be8c-b4777e953982"). InnerVolumeSpecName "kube-api-access-hjhrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:03:26.462282 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.462139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:03:26.489426 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.489390 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjhrs\" (UniqueName: \"kubernetes.io/projected/6a5c7fee-3cad-48ab-be8c-b4777e953982-kube-api-access-hjhrs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:03:26.592944 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:26.592919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:03:26.595009 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:03:26.594983 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf543a3d5_84d2_4cef_a447_23a3d684576b.slice/crio-53192fa1ee08d7fa9fff3af92b43fa43cbbe4b19d2daf9c8d14b8de3e41eba01 WatchSource:0}: Error finding container 53192fa1ee08d7fa9fff3af92b43fa43cbbe4b19d2daf9c8d14b8de3e41eba01: Status 404 returned error can't find the container with id 53192fa1ee08d7fa9fff3af92b43fa43cbbe4b19d2daf9c8d14b8de3e41eba01 Jul 27 22:03:27.242577 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.242480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqltw" event={"ID":"52efcc9e-7e46-435e-9b95-30ae552db588","Type":"ContainerStarted","Data":"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c"} Jul 27 22:03:27.242577 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.242529 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-bqltw" podUID="52efcc9e-7e46-435e-9b95-30ae552db588" containerName="authorino" containerID="cri-o://4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c" gracePeriod=30 Jul 27 22:03:27.244224 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.244195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f766b6554-lp7xx" event={"ID":"f543a3d5-84d2-4cef-a447-23a3d684576b","Type":"ContainerStarted","Data":"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd"} Jul 27 22:03:27.244341 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.244234 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f766b6554-lp7xx" event={"ID":"f543a3d5-84d2-4cef-a447-23a3d684576b","Type":"ContainerStarted","Data":"53192fa1ee08d7fa9fff3af92b43fa43cbbe4b19d2daf9c8d14b8de3e41eba01"} Jul 27 22:03:27.244341 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.244282 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc74f569d-b9cmd" Jul 27 22:03:27.274340 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.274289 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f766b6554-lp7xx" podStartSLOduration=0.98506258 podStartE2EDuration="1.27427296s" totalImagesPullingTime="289.21038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:03:27.273080972 +0000 UTC m=+527.654152890" watchObservedRunningTime="2026-07-27 22:03:27.27427296 +0000 UTC m=+527.655344884" Jul 27 22:03:27.275231 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.275202 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-bqltw" podStartSLOduration=1.869874424 podStartE2EDuration="2.275191996s" totalImagesPullingTime="405.317572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:03:27.258079726 +0000 UTC m=+527.639151651" watchObservedRunningTime="2026-07-27 22:03:27.275191996 +0000 UTC m=+527.656263920" Jul 27 22:03:27.299350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.299325 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:03:27.299504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.299477 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lf5lk" podUID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" containerName="authorino" containerID="cri-o://4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237" gracePeriod=30 Jul 27 22:03:27.317605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.317478 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cc74f569d-b9cmd"] Jul 27 22:03:27.328679 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.325319 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cc74f569d-b9cmd"] Jul 27 22:03:27.546883 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.546851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:27.560641 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.560618 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:03:27.597048 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.597016 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shbrx\" (UniqueName: \"kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx\") pod \"ecb8bac5-8c1c-438d-8a78-1b193a317b5a\" (UID: \"ecb8bac5-8c1c-438d-8a78-1b193a317b5a\") " Jul 27 22:03:27.597221 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.597063 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v2tb\" (UniqueName: \"kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb\") pod \"52efcc9e-7e46-435e-9b95-30ae552db588\" (UID: \"52efcc9e-7e46-435e-9b95-30ae552db588\") " Jul 27 22:03:27.599365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.599337 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb" (OuterVolumeSpecName: "kube-api-access-9v2tb") pod "52efcc9e-7e46-435e-9b95-30ae552db588" (UID: "52efcc9e-7e46-435e-9b95-30ae552db588"). InnerVolumeSpecName "kube-api-access-9v2tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:03:27.599469 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.599341 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx" (OuterVolumeSpecName: "kube-api-access-shbrx") pod "ecb8bac5-8c1c-438d-8a78-1b193a317b5a" (UID: "ecb8bac5-8c1c-438d-8a78-1b193a317b5a"). InnerVolumeSpecName "kube-api-access-shbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:03:27.698548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.698511 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbrx\" (UniqueName: \"kubernetes.io/projected/ecb8bac5-8c1c-438d-8a78-1b193a317b5a-kube-api-access-shbrx\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:03:27.698548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:27.698542 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v2tb\" (UniqueName: \"kubernetes.io/projected/52efcc9e-7e46-435e-9b95-30ae552db588-kube-api-access-9v2tb\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:03:28.083698 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.083664 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5c7fee-3cad-48ab-be8c-b4777e953982" path="/var/lib/kubelet/pods/6a5c7fee-3cad-48ab-be8c-b4777e953982/volumes" Jul 27 22:03:28.250091 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.250001 2559 generic.go:356] "Generic (PLEG): container finished" podID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" containerID="4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237" exitCode=0 Jul 27 22:03:28.250091 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.250049 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lf5lk" Jul 27 22:03:28.250611 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.250087 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lf5lk" event={"ID":"ecb8bac5-8c1c-438d-8a78-1b193a317b5a","Type":"ContainerDied","Data":"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237"} Jul 27 22:03:28.250611 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.250136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lf5lk" event={"ID":"ecb8bac5-8c1c-438d-8a78-1b193a317b5a","Type":"ContainerDied","Data":"be3677bec2d12a05a7d007ab503b39f4e8d30124aa3c7cd1d93076dfd2507a51"} Jul 27 22:03:28.250611 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.250175 2559 scope.go:117] "RemoveContainer" containerID="4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237" Jul 27 22:03:28.251703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.251469 2559 generic.go:356] "Generic (PLEG): container finished" podID="52efcc9e-7e46-435e-9b95-30ae552db588" containerID="4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c" exitCode=0 Jul 27 22:03:28.251703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.251519 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqltw" Jul 27 22:03:28.251703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.251554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqltw" event={"ID":"52efcc9e-7e46-435e-9b95-30ae552db588","Type":"ContainerDied","Data":"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c"} Jul 27 22:03:28.251703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.251600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqltw" event={"ID":"52efcc9e-7e46-435e-9b95-30ae552db588","Type":"ContainerDied","Data":"7d81ca75dc5761a7efe1c607729022294982b16ce9a2a99f04d566e2f8174be3"} Jul 27 22:03:28.259436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.259420 2559 scope.go:117] "RemoveContainer" containerID="4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237" Jul 27 22:03:28.259709 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:03:28.259690 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237\": container with ID starting with 4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237 not found: ID does not exist" containerID="4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237" Jul 27 22:03:28.259765 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.259717 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237"} err="failed to get container status \"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237\": rpc error: code = NotFound desc = could not find container \"4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237\": container with ID starting with 4845e64475aa667fdee9d35b49cbd7ad3f798c9e0fc285bddc5ea79e1fff1237 not found: ID does not exist" Jul 27 22:03:28.259765 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.259748 2559 scope.go:117] "RemoveContainer" containerID="4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c" Jul 27 22:03:28.268651 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.268589 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:03:28.268722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.268682 2559 scope.go:117] "RemoveContainer" containerID="4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c" Jul 27 22:03:28.268963 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:03:28.268945 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c\": container with ID starting with 4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c not found: ID does not exist" containerID="4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c" Jul 27 22:03:28.269008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.268971 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c"} err="failed to get container status \"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c\": rpc error: code = NotFound desc = could not find container \"4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c\": container with ID starting with 4c903f3157f7f9fd05f422d33d1042a4e4078e92bf63dc779d841d130c074e0c not found: ID does not exist" Jul 27 22:03:28.273773 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.273754 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lf5lk"] Jul 27 22:03:28.283307 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.283286 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:28.289342 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:28.289322 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqltw"] Jul 27 22:03:30.084055 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:30.084023 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52efcc9e-7e46-435e-9b95-30ae552db588" path="/var/lib/kubelet/pods/52efcc9e-7e46-435e-9b95-30ae552db588/volumes" Jul 27 22:03:30.084426 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:30.084372 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" path="/var/lib/kubelet/pods/ecb8bac5-8c1c-438d-8a78-1b193a317b5a/volumes" Jul 27 22:03:31.903846 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:31.903747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:32.001946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:32.001914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:32.098358 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:32.098322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:35.136107 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:35.136076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:35.235413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:35.235378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:38.282882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:38.282848 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:50.770254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:50.770203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:50.871830 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:03:50.871794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.030913 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:07.030876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.129447 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:07.129415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.716871 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:07.716834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.917626 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:07.917589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:08.626094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:08.626059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:08.726811 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:08.726778 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:12.846761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:12.846729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:16.474478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.474440 2559 generic.go:356] "Generic (PLEG): container finished" podID="96892165-bb45-4963-98fd-084a637f2b7d" containerID="bac274339fe20123226234650893438c816e719c6e8bd19412f2abc7d98de1fe" exitCode=0 Jul 27 22:04:16.474897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.474512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" event={"ID":"96892165-bb45-4963-98fd-084a637f2b7d","Type":"ContainerDied","Data":"bac274339fe20123226234650893438c816e719c6e8bd19412f2abc7d98de1fe"} Jul 27 22:04:16.474897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.474880 2559 scope.go:117] "RemoveContainer" containerID="bac274339fe20123226234650893438c816e719c6e8bd19412f2abc7d98de1fe" Jul 27 22:04:16.475883 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.475855 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:04:16.475960 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.475929 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:04:16.475994 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:16.475980 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:04:17.484464 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:17.484419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" event={"ID":"96892165-bb45-4963-98fd-084a637f2b7d","Type":"ContainerStarted","Data":"38e6aa61e6b48e182a39b272441cbd5f29adf951b6e5d42466560cf54ab18119"} Jul 27 22:04:18.120196 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:18.120132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:18.219594 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:18.219556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:18.280391 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:18.280308 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:04:18.281807 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:18.281479 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rg9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 27 22:04:18.281807 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:18.281535 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" podUID="96892165-bb45-4963-98fd-084a637f2b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 27 22:04:19.280055 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:19.280015 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rg9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 27 22:04:19.280571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:19.280087 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" podUID="96892165-bb45-4963-98fd-084a637f2b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 27 22:04:20.280297 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:20.280258 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rg9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 27 22:04:20.281311 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:20.280877 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" podUID="96892165-bb45-4963-98fd-084a637f2b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 27 22:04:21.065083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:21.065036 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:04:21.280632 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:21.280590 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rg9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 27 22:04:21.281011 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:21.280658 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" podUID="96892165-bb45-4963-98fd-084a637f2b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 27 22:04:22.283655 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:22.283624 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:04:22.284643 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:22.284626 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rg9d4" Jul 27 22:04:39.960027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:39.960001 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:04:39.960939 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:39.960917 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:04:43.692354 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:43.692315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:54.167776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:04:54.167739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:07.102504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:07.102409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:20.037258 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:20.037222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:26.568467 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:26.568430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:45.054223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054189 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh"] Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054477 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52efcc9e-7e46-435e-9b95-30ae552db588" containerName="authorino" Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054488 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52efcc9e-7e46-435e-9b95-30ae552db588" containerName="authorino" Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054508 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" containerName="authorino" Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054514 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" containerName="authorino" Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054562 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb8bac5-8c1c-438d-8a78-1b193a317b5a" containerName="authorino" Jul 27 22:05:45.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.054570 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52efcc9e-7e46-435e-9b95-30ae552db588" containerName="authorino" Jul 27 22:05:45.057336 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.057320 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.059827 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.059797 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mjxw8\"" Jul 27 22:05:45.059944 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.059829 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 22:05:45.060777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.060745 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 22:05:45.060777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.060777 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 27 22:05:45.068079 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.068055 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh"] Jul 27 22:05:45.084944 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.084913 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:45.170713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.170713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b66195-f639-4c34-a654-b1c07973da6f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.170927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/10b66195-f639-4c34-a654-b1c07973da6f-kube-api-access-vl9kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.170927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.170927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.170927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.170915 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.271583 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.271583 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b66195-f639-4c34-a654-b1c07973da6f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.271865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/10b66195-f639-4c34-a654-b1c07973da6f-kube-api-access-vl9kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.271865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.271865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.272019 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.271948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.272135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.272082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.272275 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.272185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.272347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.272327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.274036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.274009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10b66195-f639-4c34-a654-b1c07973da6f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.274266 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.274251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b66195-f639-4c34-a654-b1c07973da6f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.280613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.280589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/10b66195-f639-4c34-a654-b1c07973da6f-kube-api-access-vl9kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh\" (UID: \"10b66195-f639-4c34-a654-b1c07973da6f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.366346 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.366310 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:45.503205 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.503181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh"] Jul 27 22:05:45.505197 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:05:45.505173 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10b66195_f639_4c34_a654_b1c07973da6f.slice/crio-ba3e6845354de94a57f47deecc2d6060e9a9d6c6eeb328df242b8c875e3e99f8 WatchSource:0}: Error finding container ba3e6845354de94a57f47deecc2d6060e9a9d6c6eeb328df242b8c875e3e99f8: Status 404 returned error can't find the container with id ba3e6845354de94a57f47deecc2d6060e9a9d6c6eeb328df242b8c875e3e99f8 Jul 27 22:05:45.506419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.506404 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:05:45.897455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:45.897411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" event={"ID":"10b66195-f639-4c34-a654-b1c07973da6f","Type":"ContainerStarted","Data":"ba3e6845354de94a57f47deecc2d6060e9a9d6c6eeb328df242b8c875e3e99f8"} Jul 27 22:05:46.903586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:46.903473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" event={"ID":"10b66195-f639-4c34-a654-b1c07973da6f","Type":"ContainerStarted","Data":"7343a5235cbf6485156d3a9cc88fb8cfa105f72011dabb6dd0a8a6123a3e4581"} Jul 27 22:05:46.921504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:46.921457 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" podStartSLOduration=0.779339111 podStartE2EDuration="1.921439431s" totalImagesPullingTime="1.14210032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:05:46.919716883 +0000 UTC m=+667.300788807" watchObservedRunningTime="2026-07-27 22:05:46.921439431 +0000 UTC m=+667.302511354" Jul 27 22:05:55.367087 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:55.367051 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:55.367087 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:55.367102 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:55.379794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:55.379767 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:05:55.959491 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:05:55.959465 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh" Jul 27 22:06:39.323237 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.323143 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6995db75bc-j72mz"] Jul 27 22:06:39.326140 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.326123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.336396 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.336373 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6995db75bc-j72mz"] Jul 27 22:06:39.380305 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.380255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-tls-cert\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.380474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.380333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgppk\" (UniqueName: \"kubernetes.io/projected/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-kube-api-access-lgppk\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.481383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.481324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-tls-cert\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.481582 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.481400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgppk\" (UniqueName: \"kubernetes.io/projected/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-kube-api-access-lgppk\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.483934 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.483913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-tls-cert\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.488963 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.488942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgppk\" (UniqueName: \"kubernetes.io/projected/c1aeb91a-d48a-46ea-b5e0-aed2209630a3-kube-api-access-lgppk\") pod \"authorino-6995db75bc-j72mz\" (UID: \"c1aeb91a-d48a-46ea-b5e0-aed2209630a3\") " pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.635654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.635613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6995db75bc-j72mz" Jul 27 22:06:39.648600 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.648551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:39.745646 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.745612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:39.772244 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:39.772220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6995db75bc-j72mz"] Jul 27 22:06:39.774405 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:06:39.774373 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1aeb91a_d48a_46ea_b5e0_aed2209630a3.slice/crio-afce4107436f9c6697a7caddff874211a5b819eaadedbb365763cca100372606 WatchSource:0}: Error finding container afce4107436f9c6697a7caddff874211a5b819eaadedbb365763cca100372606: Status 404 returned error can't find the container with id afce4107436f9c6697a7caddff874211a5b819eaadedbb365763cca100372606 Jul 27 22:06:40.154405 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:40.154370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6995db75bc-j72mz" event={"ID":"c1aeb91a-d48a-46ea-b5e0-aed2209630a3","Type":"ContainerStarted","Data":"afce4107436f9c6697a7caddff874211a5b819eaadedbb365763cca100372606"} Jul 27 22:06:40.820501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:40.820406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:40.923329 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:40.923266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:41.161246 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.161207 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6995db75bc-j72mz" event={"ID":"c1aeb91a-d48a-46ea-b5e0-aed2209630a3","Type":"ContainerStarted","Data":"be209166c111ab24b6beea588950c2169c743834fadd2bfc3e4b45ac29ca3d7a"} Jul 27 22:06:41.184057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.183997 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6995db75bc-j72mz" podStartSLOduration=1.7392275879999999 podStartE2EDuration="2.183979097s" totalImagesPullingTime="444.751509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:06:41.182669478 +0000 UTC m=+721.563741403" watchObservedRunningTime="2026-07-27 22:06:41.183979097 +0000 UTC m=+721.565051027" Jul 27 22:06:41.223304 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.223270 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:06:41.223522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.223497 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f766b6554-lp7xx" podUID="f543a3d5-84d2-4cef-a447-23a3d684576b" containerName="authorino" containerID="cri-o://083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd" gracePeriod=30 Jul 27 22:06:41.496059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.496030 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:06:41.600824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.600786 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert\") pod \"f543a3d5-84d2-4cef-a447-23a3d684576b\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " Jul 27 22:06:41.600986 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.600901 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxbgb\" (UniqueName: \"kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb\") pod \"f543a3d5-84d2-4cef-a447-23a3d684576b\" (UID: \"f543a3d5-84d2-4cef-a447-23a3d684576b\") " Jul 27 22:06:41.603002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.602970 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb" (OuterVolumeSpecName: "kube-api-access-hxbgb") pod "f543a3d5-84d2-4cef-a447-23a3d684576b" (UID: "f543a3d5-84d2-4cef-a447-23a3d684576b"). InnerVolumeSpecName "kube-api-access-hxbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:06:41.610609 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.610583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f543a3d5-84d2-4cef-a447-23a3d684576b" (UID: "f543a3d5-84d2-4cef-a447-23a3d684576b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:06:41.701896 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.701864 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbgb\" (UniqueName: \"kubernetes.io/projected/f543a3d5-84d2-4cef-a447-23a3d684576b-kube-api-access-hxbgb\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:06:41.701896 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:41.701893 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f543a3d5-84d2-4cef-a447-23a3d684576b-tls-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:06:42.167506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.167477 2559 generic.go:356] "Generic (PLEG): container finished" podID="f543a3d5-84d2-4cef-a447-23a3d684576b" containerID="083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd" exitCode=0 Jul 27 22:06:42.168006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.167526 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f766b6554-lp7xx" Jul 27 22:06:42.168006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.167575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f766b6554-lp7xx" event={"ID":"f543a3d5-84d2-4cef-a447-23a3d684576b","Type":"ContainerDied","Data":"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd"} Jul 27 22:06:42.168006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.167630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f766b6554-lp7xx" event={"ID":"f543a3d5-84d2-4cef-a447-23a3d684576b","Type":"ContainerDied","Data":"53192fa1ee08d7fa9fff3af92b43fa43cbbe4b19d2daf9c8d14b8de3e41eba01"} Jul 27 22:06:42.168006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.167655 2559 scope.go:117] "RemoveContainer" containerID="083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd" Jul 27 22:06:42.176240 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.176220 2559 scope.go:117] "RemoveContainer" containerID="083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd" Jul 27 22:06:42.176529 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:06:42.176508 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd\": container with ID starting with 083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd not found: ID does not exist" containerID="083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd" Jul 27 22:06:42.176586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.176539 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd"} err="failed to get container status \"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd\": rpc error: code = NotFound desc = could not find container \"083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd\": container with ID starting with 083c68172a34dc387449f6f47d9eac2135c13df448908afc64657977e0ac83bd not found: ID does not exist" Jul 27 22:06:42.194292 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.194260 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:06:42.210456 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:42.210429 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f766b6554-lp7xx"] Jul 27 22:06:44.083104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:44.083064 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f543a3d5-84d2-4cef-a447-23a3d684576b" path="/var/lib/kubelet/pods/f543a3d5-84d2-4cef-a447-23a3d684576b/volumes" Jul 27 22:06:58.158986 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:06:58.158943 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:11.215430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:07:11.215392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:11.316050 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:07:11.315998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:21.597824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:07:21.597786 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:21.671461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:07:21.671402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:07.227619 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:08:07.227522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:44.357854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:08:44.357816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:45.616236 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:08:45.616201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:19.105859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:19.105827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:19.218184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:19.218123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:19.316765 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:19.316708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:25.428724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:25.428683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:25.501921 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:25.501874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:25.597307 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:25.597266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:25.804275 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:25.804183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:31.996517 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:31.996416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:32.094541 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:32.094511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:39.717104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:39.717061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:39.988520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:39.988431 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:09:39.990825 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:09:39.990804 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:12:43.354449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:12:43.354352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:44.562605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:12:44.562575 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:48.776821 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:12:48.776783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:40.022276 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:14:40.022247 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:14:40.024756 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:14:40.024652 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:15:00.164522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.164487 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t"] Jul 27 22:15:00.164964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.164778 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f543a3d5-84d2-4cef-a447-23a3d684576b" containerName="authorino" Jul 27 22:15:00.164964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.164789 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f543a3d5-84d2-4cef-a447-23a3d684576b" containerName="authorino" Jul 27 22:15:00.164964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.164837 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f543a3d5-84d2-4cef-a447-23a3d684576b" containerName="authorino" Jul 27 22:15:00.167764 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.167748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" Jul 27 22:15:00.170501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.170487 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:15:00.178374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.178348 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" Jul 27 22:15:00.178792 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.178769 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t"] Jul 27 22:15:00.320703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.320678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t"] Jul 27 22:15:00.323051 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:15:00.323027 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1beba94_4b34_4fb4_b77a_ff48d2d02249.slice/crio-aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e WatchSource:0}: Error finding container aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e: Status 404 returned error can't find the container with id aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e Jul 27 22:15:00.324904 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.324888 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:15:00.519695 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:00.519608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" event={"ID":"d1beba94-4b34-4fb4-b77a-ff48d2d02249","Type":"ContainerStarted","Data":"aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e"} Jul 27 22:15:03.304557 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:03.304527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:03.537029 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:03.536992 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" event={"ID":"d1beba94-4b34-4fb4-b77a-ff48d2d02249","Type":"ContainerStarted","Data":"c54b489ca739322935dbfda6a3319d615d6d87562e13d3a87596d500de967cb0"} Jul 27 22:15:03.556183 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:03.556036 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" podStartSLOduration=1.194779116 podStartE2EDuration="3.556015572s" totalImagesPullingTime="2.361236456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:15:03.555137349 +0000 UTC m=+1223.936209273" watchObservedRunningTime="2026-07-27 22:15:03.556015572 +0000 UTC m=+1223.937087497" Jul 27 22:15:04.543537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:04.543500 2559 generic.go:356] "Generic (PLEG): container finished" podID="d1beba94-4b34-4fb4-b77a-ff48d2d02249" containerID="c54b489ca739322935dbfda6a3319d615d6d87562e13d3a87596d500de967cb0" exitCode=0 Jul 27 22:15:04.543890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:04.543601 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" event={"ID":"d1beba94-4b34-4fb4-b77a-ff48d2d02249","Type":"ContainerDied","Data":"c54b489ca739322935dbfda6a3319d615d6d87562e13d3a87596d500de967cb0"} Jul 27 22:15:05.673103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:05.673080 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" Jul 27 22:15:06.553695 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:06.553669 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" Jul 27 22:15:06.553885 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:06.553705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753175-9fv9t" event={"ID":"d1beba94-4b34-4fb4-b77a-ff48d2d02249","Type":"ContainerDied","Data":"aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e"} Jul 27 22:15:06.553885 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:15:06.553755 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf193e1d161c49a4a640dc285d808955420b924dc828ea99a6ef536db47ce6e" Jul 27 22:17:22.195952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:22.195857 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:25.552969 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.552937 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:17:25.553360 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.553267 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1beba94-4b34-4fb4-b77a-ff48d2d02249" containerName="cleanup" Jul 27 22:17:25.553360 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.553279 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1beba94-4b34-4fb4-b77a-ff48d2d02249" containerName="cleanup" Jul 27 22:17:25.553360 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.553353 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1beba94-4b34-4fb4-b77a-ff48d2d02249" containerName="cleanup" Jul 27 22:17:25.555832 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.555812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.559017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.558996 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-cc5625f3-kserve-self-signed-certs\"" Jul 27 22:17:25.578837 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.578800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:17:25.596551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.596517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:25.624773 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.624773 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.625027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.625027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.625027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.625027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.624951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvkn\" (UniqueName: \"kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvkn\" (UniqueName: \"kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.725965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.725942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.726228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.726200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.726453 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.726375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.728128 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.728107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.728400 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.728383 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.734103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.734076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvkn\" (UniqueName: \"kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn\") pod \"e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:25.865106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:25.865068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:26.018449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:26.018417 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:17:26.217655 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:26.217558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" event={"ID":"cbf35372-4026-44fe-b15b-0ce592a19ad6","Type":"ContainerStarted","Data":"62e285f8303efa9c5dd7765bad22617f465703480247be80bc035cb4bb5fed48"} Jul 27 22:17:26.217655 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:26.217610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" event={"ID":"cbf35372-4026-44fe-b15b-0ce592a19ad6","Type":"ContainerStarted","Data":"4634019564eaf26096b9d510841f07839684a9c6d76e4366b011c266e6e604e3"} Jul 27 22:17:26.247439 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:26.247386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" podStartSLOduration=1.247366832 podStartE2EDuration="1.247366832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:17:26.246677349 +0000 UTC m=+1366.627749272" watchObservedRunningTime="2026-07-27 22:17:26.247366832 +0000 UTC m=+1366.628438755" Jul 27 22:17:35.865935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:35.865888 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:35.866456 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:35.865951 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:35.886350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:35.886319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:17:36.283080 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:17:36.283000 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:18:13.766001 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:13.765964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:14.344491 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:14.344458 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:18:14.846963 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:14.846933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:18:29.364018 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.363971 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" podUID="cbf35372-4026-44fe-b15b-0ce592a19ad6" containerName="main" containerID="cri-o://62e285f8303efa9c5dd7765bad22617f465703480247be80bc035cb4bb5fed48" gracePeriod=45 Jul 27 22:18:29.526794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.526761 2559 generic.go:356] "Generic (PLEG): container finished" podID="cbf35372-4026-44fe-b15b-0ce592a19ad6" containerID="62e285f8303efa9c5dd7765bad22617f465703480247be80bc035cb4bb5fed48" exitCode=0 Jul 27 22:18:29.526960 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.526837 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" event={"ID":"cbf35372-4026-44fe-b15b-0ce592a19ad6","Type":"ContainerDied","Data":"62e285f8303efa9c5dd7765bad22617f465703480247be80bc035cb4bb5fed48"} Jul 27 22:18:29.608361 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.608332 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:18:29.669758 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669671 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.669758 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.669974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669784 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flvkn\" (UniqueName: \"kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.669974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669819 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.669974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.669974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.669890 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir\") pod \"cbf35372-4026-44fe-b15b-0ce592a19ad6\" (UID: \"cbf35372-4026-44fe-b15b-0ce592a19ad6\") " Jul 27 22:18:29.670212 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.670003 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:18:29.670212 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.670095 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home" (OuterVolumeSpecName: "home") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:18:29.670321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.670222 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:18:29.670321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.670235 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-model-cache\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:29.670321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.670257 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-home\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:29.672137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.672116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm" (OuterVolumeSpecName: "dshm") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:18:29.672255 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.672203 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:18:29.672255 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.672245 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn" (OuterVolumeSpecName: "kube-api-access-flvkn") pod "cbf35372-4026-44fe-b15b-0ce592a19ad6" (UID: "cbf35372-4026-44fe-b15b-0ce592a19ad6"). InnerVolumeSpecName "kube-api-access-flvkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:18:29.770977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.770934 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flvkn\" (UniqueName: \"kubernetes.io/projected/cbf35372-4026-44fe-b15b-0ce592a19ad6-kube-api-access-flvkn\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:29.770977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.770970 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf35372-4026-44fe-b15b-0ce592a19ad6-tls-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:29.770977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.770980 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-tmp-dir\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:29.770977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:29.770987 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf35372-4026-44fe-b15b-0ce592a19ad6-dshm\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:18:30.532163 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:30.532119 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" event={"ID":"cbf35372-4026-44fe-b15b-0ce592a19ad6","Type":"ContainerDied","Data":"4634019564eaf26096b9d510841f07839684a9c6d76e4366b011c266e6e604e3"} Jul 27 22:18:30.532543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:30.532184 2559 scope.go:117] "RemoveContainer" containerID="62e285f8303efa9c5dd7765bad22617f465703480247be80bc035cb4bb5fed48" Jul 27 22:18:30.532543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:30.532127 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb" Jul 27 22:18:30.549496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:30.549473 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:18:30.555624 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:30.555596 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-cc5625f3-kserve-7655747bc5-cr9bb"] Jul 27 22:18:32.088860 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:18:32.088825 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf35372-4026-44fe-b15b-0ce592a19ad6" path="/var/lib/kubelet/pods/cbf35372-4026-44fe-b15b-0ce592a19ad6/volumes" Jul 27 22:19:40.056532 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:19:40.056405 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:19:40.060555 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:19:40.058825 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:19:44.480711 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:19:44.480678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:44.533170 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:19:44.533116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:50.783736 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:19:50.783700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:00.821479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:20:00.821399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:22.495262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:22.495232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:44.001993 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:44.001903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:47.126417 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:47.126379 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:47.238637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:47.238609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:47.576044 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:47.575939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:53.199357 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.199301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:53.267413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.267382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:53.440204 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.440141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:53.739876 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.739840 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:21:53.740292 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.740275 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf35372-4026-44fe-b15b-0ce592a19ad6" containerName="main" Jul 27 22:21:53.740389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.740295 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf35372-4026-44fe-b15b-0ce592a19ad6" containerName="main" Jul 27 22:21:53.740444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.740393 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf35372-4026-44fe-b15b-0ce592a19ad6" containerName="main" Jul 27 22:21:53.743734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.743711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.746576 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.746557 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b8ed72f2-openshift-default-dockercfg-smc86\"" Jul 27 22:21:53.768193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.768163 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:21:53.777395 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.777370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:53.832537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.832793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.833120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.832796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk645\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.933847 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.933805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934030 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.933862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934030 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.933900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934132 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934132 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk645\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934542 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.934851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.934830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.936495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.936474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.936730 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.936710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.943473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.943454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:53.943597 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:53.943550 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk645\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645\") pod \"e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:54.015332 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.015244 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:54.053036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.052997 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:54.192294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.192271 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:21:54.194248 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:21:54.194219 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35095d4c_f4ac_4760_a27a_6b04b410f264.slice/crio-78e76246abe7cbef2f958cc604aa4b9512f3ced05fecb77eb711aa1309255c7a WatchSource:0}: Error finding container 78e76246abe7cbef2f958cc604aa4b9512f3ced05fecb77eb711aa1309255c7a: Status 404 returned error can't find the container with id 78e76246abe7cbef2f958cc604aa4b9512f3ced05fecb77eb711aa1309255c7a Jul 27 22:21:54.196558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.196530 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:21:54.196647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.196596 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:21:54.196647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.196624 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:21:54.494922 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.494881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerStarted","Data":"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01"} Jul 27 22:21:54.495328 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.494941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerStarted","Data":"78e76246abe7cbef2f958cc604aa4b9512f3ced05fecb77eb711aa1309255c7a"} Jul 27 22:21:54.518556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:54.518505 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podStartSLOduration=1.518490351 podStartE2EDuration="1.518490351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:21:54.516028809 +0000 UTC m=+1634.897100733" watchObservedRunningTime="2026-07-27 22:21:54.518490351 +0000 UTC m=+1634.899562324" Jul 27 22:21:55.053351 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:55.053313 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:55.058024 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:55.057999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:55.500165 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:55.500125 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:55.501222 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:55.501205 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:21:55.892273 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:55.892240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.007379 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:56.007349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.112312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:56.112277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.601648 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:56.601607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.735082 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:56.735047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:56.832162 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:21:56.832108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:07.559084 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.559048 2559 generic.go:356] "Generic (PLEG): container finished" podID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerID="f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01" exitCode=0 Jul 27 22:22:07.559570 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.559141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerDied","Data":"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01"} Jul 27 22:22:07.559657 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.559603 2559 scope.go:117] "RemoveContainer" containerID="f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01" Jul 27 22:22:07.560899 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.560866 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:07.561008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.560957 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:07.561058 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:07.561009 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:08.565494 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:08.565456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerStarted","Data":"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb"} Jul 27 22:22:09.053864 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:09.053818 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:09.054975 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:09.054952 2559 patch_prober.go:28] interesting pod/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 27 22:22:09.055034 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:09.055006 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 27 22:22:10.054099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:10.054061 2559 patch_prober.go:28] interesting pod/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 27 22:22:10.054544 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:10.054122 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 27 22:22:10.500082 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:10.500035 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:11.053981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:11.053947 2559 patch_prober.go:28] interesting pod/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 27 22:22:11.054193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:11.054019 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 27 22:22:12.054261 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:12.054221 2559 patch_prober.go:28] interesting pod/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 27 22:22:12.054654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:12.054296 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 27 22:22:13.058023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:13.057994 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:13.059114 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:13.059099 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:32.117434 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:32.117401 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:32.222404 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:32.222366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:32.320917 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:32.320876 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:36.582215 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:36.582142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:36.727040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:36.727008 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:22:36.727266 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:36.727243 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" containerID="cri-o://943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb" gracePeriod=30 Jul 27 22:22:36.764328 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:36.764283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.002462 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.002413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.275642 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.275543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.361696 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.361665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.489676 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.489642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.787262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.787233 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:22:37.790640 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.790621 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.794052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.794032 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-732c17ff-gw-openshift-default-dockercfg-k4wnn\"" Jul 27 22:22:37.804136 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.804113 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:22:37.820204 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.820178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:37.921367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpkdt\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.921829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.921688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:37.928488 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:37.928460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:38.022848 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpkdt\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.022967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023505 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023723 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.023782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.023765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.025360 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.025337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.025502 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.025484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.031626 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.031600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.031743 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.031726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpkdt\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt\") pod \"e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.100486 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.100408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:38.238663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.238628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:22:38.241659 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:22:38.241636 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99ff5e4_7728_409d_a7b3_ce5a578b51bd.slice/crio-5976bebcf940ec4c380b526d552d784804a43c15e6586194949ee3df57f3023d WatchSource:0}: Error finding container 5976bebcf940ec4c380b526d552d784804a43c15e6586194949ee3df57f3023d: Status 404 returned error can't find the container with id 5976bebcf940ec4c380b526d552d784804a43c15e6586194949ee3df57f3023d Jul 27 22:22:38.243795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.243760 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:38.243897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.243838 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:38.243897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.243876 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:38.708970 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.708930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerStarted","Data":"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8"} Jul 27 22:22:38.709143 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.708994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerStarted","Data":"5976bebcf940ec4c380b526d552d784804a43c15e6586194949ee3df57f3023d"} Jul 27 22:22:38.732955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:38.732884 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podStartSLOduration=1.732865505 podStartE2EDuration="1.732865505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:22:38.729352816 +0000 UTC m=+1679.110424741" watchObservedRunningTime="2026-07-27 22:22:38.732865505 +0000 UTC m=+1679.113937429" Jul 27 22:22:39.101388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:39.101291 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:39.106341 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:39.106314 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:39.713659 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:39.713626 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:39.714613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:39.714596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:40.202471 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:40.202429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:40.323015 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:40.322980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:40.446555 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:40.446519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:40.843500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:40.843469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:40.943640 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:40.943593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:41.979848 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:41.979818 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:42.059274 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059274 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059297 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059350 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059515 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk645\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket\") pod \"35095d4c-f4ac-4760-a27a-6b04b410f264\" (UID: \"35095d4c-f4ac-4760-a27a-6b04b410f264\") " Jul 27 22:22:42.059848 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059745 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:42.059903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059848 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:42.059977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.059955 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:42.060092 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060049 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:22:42.060228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060209 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35095d4c-f4ac-4760-a27a-6b04b410f264-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.060288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060235 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.060288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060250 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.060288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060263 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.060397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.060292 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data" (OuterVolumeSpecName: "istio-data") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:42.061911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.061886 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:22:42.061911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.061888 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:22:42.062236 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.062217 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token" (OuterVolumeSpecName: "istio-token") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:42.062314 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.062271 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645" (OuterVolumeSpecName: "kube-api-access-vk645") pod "35095d4c-f4ac-4760-a27a-6b04b410f264" (UID: "35095d4c-f4ac-4760-a27a-6b04b410f264"). InnerVolumeSpecName "kube-api-access-vk645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:22:42.160984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.160941 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.160984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.160973 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.160984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.160982 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.160984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.160989 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35095d4c-f4ac-4760-a27a-6b04b410f264-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.160984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.160998 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk645\" (UniqueName: \"kubernetes.io/projected/35095d4c-f4ac-4760-a27a-6b04b410f264-kube-api-access-vk645\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:22:42.729822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.729786 2559 generic.go:356] "Generic (PLEG): container finished" podID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerID="943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb" exitCode=0 Jul 27 22:22:42.730033 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.729843 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" Jul 27 22:22:42.730033 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.729853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerDied","Data":"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb"} Jul 27 22:22:42.730033 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.729900 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25" event={"ID":"35095d4c-f4ac-4760-a27a-6b04b410f264","Type":"ContainerDied","Data":"78e76246abe7cbef2f958cc604aa4b9512f3ced05fecb77eb711aa1309255c7a"} Jul 27 22:22:42.730033 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.729920 2559 scope.go:117] "RemoveContainer" containerID="943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb" Jul 27 22:22:42.739014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.738995 2559 scope.go:117] "RemoveContainer" containerID="f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01" Jul 27 22:22:42.747754 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.747731 2559 scope.go:117] "RemoveContainer" containerID="943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb" Jul 27 22:22:42.748072 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:22:42.748052 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb\": container with ID starting with 943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb not found: ID does not exist" containerID="943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb" Jul 27 22:22:42.748140 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.748082 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb"} err="failed to get container status \"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb\": rpc error: code = NotFound desc = could not find container \"943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb\": container with ID starting with 943d456bf73e0ead75c4ad901e1ceb581e78c68bca7453915d1e9c65efbfa8eb not found: ID does not exist" Jul 27 22:22:42.748140 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.748103 2559 scope.go:117] "RemoveContainer" containerID="f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01" Jul 27 22:22:42.748363 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:22:42.748347 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01\": container with ID starting with f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01 not found: ID does not exist" containerID="f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01" Jul 27 22:22:42.748421 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.748366 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01"} err="failed to get container status \"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01\": rpc error: code = NotFound desc = could not find container \"f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01\": container with ID starting with f3f1630fb8d3964e1b0c80f38839da5d878e8098a38d17b230a1371b0a8e3b01 not found: ID does not exist" Jul 27 22:22:42.748724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.748702 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:22:42.752291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:42.752262 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b8ed72f2-openshift-default-6544cbb844-lml25"] Jul 27 22:22:44.084062 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:44.084027 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" path="/var/lib/kubelet/pods/35095d4c-f4ac-4760-a27a-6b04b410f264/volumes" Jul 27 22:22:54.715338 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.715262 2559 patch_prober.go:28] interesting pod/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 27 22:22:54.715338 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.715320 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 27 22:22:54.793714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.793684 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerID="24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8" exitCode=0 Jul 27 22:22:54.793880 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.793761 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerDied","Data":"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8"} Jul 27 22:22:54.794202 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.794169 2559 scope.go:117] "RemoveContainer" containerID="24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8" Jul 27 22:22:54.795105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.795080 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:54.795179 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.795166 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:54.795228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:54.795198 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:22:55.803317 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:55.803280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerStarted","Data":"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd"} Jul 27 22:22:56.100833 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:56.100725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:22:56.101044 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:56.101017 2559 patch_prober.go:28] interesting pod/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 27 22:22:56.101110 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:56.101078 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 27 22:22:57.101585 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:57.101541 2559 patch_prober.go:28] interesting pod/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 27 22:22:57.101971 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:57.101614 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 27 22:22:58.101760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:58.101720 2559 patch_prober.go:28] interesting pod/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 27 22:22:58.102313 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:58.101791 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 27 22:22:59.101555 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:59.101509 2559 patch_prober.go:28] interesting pod/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 27 22:22:59.101755 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:22:59.101580 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 27 22:23:00.105005 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:00.104971 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:23:00.105437 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:00.105416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:23:00.105927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:00.105913 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:23:05.267475 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.267438 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2"] Jul 27 22:23:05.267882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.267786 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:05.267882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.267799 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:05.267882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.267874 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:05.271292 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.271273 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:05.274080 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.274055 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:23:05.274234 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.274058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:23:05.279898 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.279873 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2"] Jul 27 22:23:05.363567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.363527 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2\" (UID: \"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:05.464353 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.464306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2\" (UID: \"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:05.464950 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.464933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2\" (UID: \"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:05.582977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.582892 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:05.720753 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.720726 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2"] Jul 27 22:23:05.722837 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:23:05.722804 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84c1ded_8c1e_47ff_b7b3_e6a9bfbd7ea8.slice/crio-75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348 WatchSource:0}: Error finding container 75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348: Status 404 returned error can't find the container with id 75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348 Jul 27 22:23:05.851246 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:05.851159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" event={"ID":"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8","Type":"ContainerStarted","Data":"75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348"} Jul 27 22:23:06.857897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:06.857861 2559 generic.go:356] "Generic (PLEG): container finished" podID="b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" containerID="60d57fdb77bee243cdadb8c388e4f48f568396be0279c697a195bcc9984273ed" exitCode=0 Jul 27 22:23:06.858319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:06.857944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" event={"ID":"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8","Type":"ContainerDied","Data":"60d57fdb77bee243cdadb8c388e4f48f568396be0279c697a195bcc9984273ed"} Jul 27 22:23:07.991689 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:07.991663 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:08.086228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.086201 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca\") pod \"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8\" (UID: \"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8\") " Jul 27 22:23:08.086516 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.086495 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" (UID: "b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:23:08.187042 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.186954 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:08.629737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.629705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:08.868934 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.868906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" event={"ID":"b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8","Type":"ContainerDied","Data":"75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348"} Jul 27 22:23:08.869108 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.868956 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75be69c192e949ee28c97ce95f5c54b7c9654aeffd77cf68ecad16ca4722f348" Jul 27 22:23:08.869108 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:08.868912 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2" Jul 27 22:23:15.756174 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:15.756119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:16.917290 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:16.917253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:20.339915 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.339878 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:20.347544 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.347518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2"] Jul 27 22:23:20.351952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.351932 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-732c17ff-p85t2"] Jul 27 22:23:20.490029 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.490000 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:23:20.490280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.490239 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" containerID="cri-o://020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd" gracePeriod=30 Jul 27 22:23:20.517060 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.517030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:20.754777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.754741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:20.985031 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:20.984998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:21.297389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297361 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:23:21.297674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297662 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:21.297719 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297675 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:21.297719 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297685 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" containerName="revoke-keys" Jul 27 22:23:21.297719 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297690 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" containerName="revoke-keys" Jul 27 22:23:21.297812 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297741 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" containerName="revoke-keys" Jul 27 22:23:21.297812 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.297749 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35095d4c-f4ac-4760-a27a-6b04b410f264" containerName="istio-proxy" Jul 27 22:23:21.300733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.300713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.302993 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.302972 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-294097f5-openshift-default-dockercfg-r49q4\"" Jul 27 22:23:21.312713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.312691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:23:21.329444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.329419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:21.404339 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbxq\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.404796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.404702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbxq\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.505772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.506102 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.505932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.506102 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.506010 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.506198 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.506104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.506276 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.506242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.506408 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.506366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.508088 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.508069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.508326 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.508310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.513691 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.513666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.513788 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.513716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbxq\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq\") pod \"e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.610830 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.610788 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:21.769933 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.769908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:23:21.772476 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:23:21.772444 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354455e4_a80a_4889_9c64_12b71a02c375.slice/crio-3fd1c756adff3e45c7cbb31477312f72d91c77c6aed0c734c7aa1dab22e0616c WatchSource:0}: Error finding container 3fd1c756adff3e45c7cbb31477312f72d91c77c6aed0c734c7aa1dab22e0616c: Status 404 returned error can't find the container with id 3fd1c756adff3e45c7cbb31477312f72d91c77c6aed0c734c7aa1dab22e0616c Jul 27 22:23:21.775118 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.775085 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:21.775203 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.775168 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:21.775247 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.775199 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:21.786673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.786641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:21.942423 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.942314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerStarted","Data":"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a"} Jul 27 22:23:21.942423 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.942411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerStarted","Data":"3fd1c756adff3e45c7cbb31477312f72d91c77c6aed0c734c7aa1dab22e0616c"} Jul 27 22:23:21.966550 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:21.966483 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podStartSLOduration=0.966462852 podStartE2EDuration="966.462852ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:23:21.96282739 +0000 UTC m=+1722.343899314" watchObservedRunningTime="2026-07-27 22:23:21.966462852 +0000 UTC m=+1722.347534782" Jul 27 22:23:22.084220 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:22.084185 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8" path="/var/lib/kubelet/pods/b84c1ded-8c1e-47ff-b7b3-e6a9bfbd7ea8/volumes" Jul 27 22:23:22.611654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:22.611615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:22.616548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:22.616524 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:22.947940 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:22.947843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:22.948901 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:22.948878 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:23.308461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:23.308361 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:23.389573 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:23.389543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:23.462095 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:23.462067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:23.561935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:23.561796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:24.002608 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:24.002573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:24.094701 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:24.094668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:25.735162 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.735126 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:23:25.846415 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846415 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846370 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpkdt\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846415 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846389 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846422 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846448 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846471 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846503 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846556 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846675 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846585 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token\") pod \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\" (UID: \"e99ff5e4-7728-409d-a7b3-ce5a578b51bd\") " Jul 27 22:23:25.846985 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846838 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:23:25.846985 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.846961 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:25.847098 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.847028 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:25.847323 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.847256 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:25.847483 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.847461 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:25.848759 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.848733 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt" (OuterVolumeSpecName: "kube-api-access-dpkdt") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "kube-api-access-dpkdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:25.848855 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.848819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:23:25.849338 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.849317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:25.849411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.849330 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e99ff5e4-7728-409d-a7b3-ce5a578b51bd" (UID: "e99ff5e4-7728-409d-a7b3-ce5a578b51bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:25.947634 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947596 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947634 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947626 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpkdt\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-kube-api-access-dpkdt\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947634 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947639 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947650 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947669 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947679 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947692 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947703 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.947891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.947714 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e99ff5e4-7728-409d-a7b3-ce5a578b51bd-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:25.964514 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.964482 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerID="020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd" exitCode=0 Jul 27 22:23:25.964639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.964531 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" Jul 27 22:23:25.964639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.964564 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerDied","Data":"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd"} Jul 27 22:23:25.964639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.964609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h" event={"ID":"e99ff5e4-7728-409d-a7b3-ce5a578b51bd","Type":"ContainerDied","Data":"5976bebcf940ec4c380b526d552d784804a43c15e6586194949ee3df57f3023d"} Jul 27 22:23:25.964639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.964635 2559 scope.go:117] "RemoveContainer" containerID="020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd" Jul 27 22:23:25.974428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.974411 2559 scope.go:117] "RemoveContainer" containerID="24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8" Jul 27 22:23:25.982416 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.982399 2559 scope.go:117] "RemoveContainer" containerID="020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd" Jul 27 22:23:25.982657 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:23:25.982639 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd\": container with ID starting with 020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd not found: ID does not exist" containerID="020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd" Jul 27 22:23:25.982729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.982673 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd"} err="failed to get container status \"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd\": rpc error: code = NotFound desc = could not find container \"020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd\": container with ID starting with 020e84ff157ba0f22021c472421a5f4c5b4d2abb62d77dc00348f962c569cdfd not found: ID does not exist" Jul 27 22:23:25.982729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.982698 2559 scope.go:117] "RemoveContainer" containerID="24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8" Jul 27 22:23:25.982946 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:23:25.982930 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8\": container with ID starting with 24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8 not found: ID does not exist" containerID="24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8" Jul 27 22:23:25.982996 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.982951 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8"} err="failed to get container status \"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8\": rpc error: code = NotFound desc = could not find container \"24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8\": container with ID starting with 24eb01d37c076a06d42b2df33eff69ab4021ad808e6f2d2b3bfaea84d61c16f8 not found: ID does not exist" Jul 27 22:23:25.987868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.987846 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:23:25.991798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:25.991775 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-732c17ff-gw-openshift-default-58cfb69b56-h774h"] Jul 27 22:23:26.089965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:26.089772 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" path="/var/lib/kubelet/pods/e99ff5e4-7728-409d-a7b3-ce5a578b51bd/volumes" Jul 27 22:23:37.949638 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:37.949602 2559 patch_prober.go:28] interesting pod/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 27 22:23:37.950025 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:37.949671 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 27 22:23:38.028484 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.028451 2559 generic.go:356] "Generic (PLEG): container finished" podID="354455e4-a80a-4889-9c64-12b71a02c375" containerID="6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a" exitCode=0 Jul 27 22:23:38.028648 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.028531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerDied","Data":"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a"} Jul 27 22:23:38.028940 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.028924 2559 scope.go:117] "RemoveContainer" containerID="6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a" Jul 27 22:23:38.029913 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.029884 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:38.029984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.029967 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:38.030027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:38.030011 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:39.035722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:39.035685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerStarted","Data":"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054"} Jul 27 22:23:39.611415 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:39.611375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:39.611664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:39.611636 2559 patch_prober.go:28] interesting pod/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 27 22:23:39.611732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:39.611710 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 27 22:23:40.612142 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:40.612114 2559 patch_prober.go:28] interesting pod/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 27 22:23:40.612537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:40.612197 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 27 22:23:41.611701 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:41.611662 2559 patch_prober.go:28] interesting pod/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 27 22:23:41.611887 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:41.611721 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 27 22:23:42.611605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:42.611565 2559 patch_prober.go:28] interesting pod/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 27 22:23:42.612002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:42.611662 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 27 22:23:43.615238 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:43.615208 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:43.615607 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:43.615508 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:43.616057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:43.616042 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:23:49.096105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096067 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6"] Jul 27 22:23:49.096519 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096371 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:23:49.096519 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096382 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:23:49.096519 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096395 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:23:49.096519 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096401 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:23:49.096519 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.096448 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:23:49.099360 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.099343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:49.101815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.101785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:23:49.101815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.101803 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:23:49.106175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.106128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6"] Jul 27 22:23:49.164069 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.164039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-294097f5-74qc6\" (UID: \"c9977084-fab0-44ef-a637-842dea6ee0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:49.264685 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.264649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-294097f5-74qc6\" (UID: \"c9977084-fab0-44ef-a637-842dea6ee0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:49.265564 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.265535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-294097f5-74qc6\" (UID: \"c9977084-fab0-44ef-a637-842dea6ee0f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:49.410448 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.410396 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:49.538055 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:49.538029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6"] Jul 27 22:23:49.540739 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:23:49.540708 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9977084_fab0_44ef_a637_842dea6ee0f2.slice/crio-93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98 WatchSource:0}: Error finding container 93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98: Status 404 returned error can't find the container with id 93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98 Jul 27 22:23:50.088813 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:50.088783 2559 generic.go:356] "Generic (PLEG): container finished" podID="c9977084-fab0-44ef-a637-842dea6ee0f2" containerID="5f6061255c7173aa83a1f46da171284a372ca9e5b2c07a3721acefc3030ca8db" exitCode=0 Jul 27 22:23:50.088977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:50.088841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" event={"ID":"c9977084-fab0-44ef-a637-842dea6ee0f2","Type":"ContainerDied","Data":"5f6061255c7173aa83a1f46da171284a372ca9e5b2c07a3721acefc3030ca8db"} Jul 27 22:23:50.088977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:50.088876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" event={"ID":"c9977084-fab0-44ef-a637-842dea6ee0f2","Type":"ContainerStarted","Data":"93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98"} Jul 27 22:23:51.222760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:51.222738 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:51.278262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:51.278227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca\") pod \"c9977084-fab0-44ef-a637-842dea6ee0f2\" (UID: \"c9977084-fab0-44ef-a637-842dea6ee0f2\") " Jul 27 22:23:51.278561 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:51.278539 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9977084-fab0-44ef-a637-842dea6ee0f2" (UID: "c9977084-fab0-44ef-a637-842dea6ee0f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:23:51.378935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:51.378904 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9977084-fab0-44ef-a637-842dea6ee0f2-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:23:52.102849 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:52.102819 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" Jul 27 22:23:52.102849 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:52.102832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6" event={"ID":"c9977084-fab0-44ef-a637-842dea6ee0f2","Type":"ContainerDied","Data":"93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98"} Jul 27 22:23:52.103083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:52.102876 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93808294f7f027732c45db9e87b986d7fcf45479e619a2b8af683d8e6c2c4e98" Jul 27 22:23:59.637633 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:59.637599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:59.737053 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:59.737023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:59.839040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:23:59.839001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:04.183399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:04.183365 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:04.187602 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:04.187577 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6"] Jul 27 22:24:04.190529 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:04.190506 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-294097f5-74qc6"] Jul 27 22:24:05.384878 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:05.384846 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:24:05.385387 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:05.385129 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" containerID="cri-o://aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054" gracePeriod=30 Jul 27 22:24:05.414726 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:05.414696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.640388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:05.640300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.870547 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:05.870489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.083414 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.083340 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9977084-fab0-44ef-a637-842dea6ee0f2" path="/var/lib/kubelet/pods/c9977084-fab0-44ef-a637-842dea6ee0f2/volumes" Jul 27 22:24:06.186768 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.186732 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:06.187095 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.187077 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9977084-fab0-44ef-a637-842dea6ee0f2" containerName="revoke-keys" Jul 27 22:24:06.187192 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.187097 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9977084-fab0-44ef-a637-842dea6ee0f2" containerName="revoke-keys" Jul 27 22:24:06.187265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.187219 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99ff5e4-7728-409d-a7b3-ce5a578b51bd" containerName="istio-proxy" Jul 27 22:24:06.187265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.187237 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9977084-fab0-44ef-a637-842dea6ee0f2" containerName="revoke-keys" Jul 27 22:24:06.190467 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.190445 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.193498 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.193481 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-858f23ab-openshift-default-dockercfg-qrzhc\"" Jul 27 22:24:06.208395 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.208372 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:06.232537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.232500 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.297872 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.297803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.297911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg5p\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.297942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.297974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.298003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.298102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.298140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.298270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.298349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.298307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg5p\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.399794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.400135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.400135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.400135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.399984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.400282 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.400188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.401861 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.401836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.402114 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.402097 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.413795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.413774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.414060 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.414039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg5p\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p\") pod \"e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.483345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.483306 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:06.500529 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.500498 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:06.642647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.642624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:06.644665 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:24:06.644642 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f20593d_2699_4598_8e0e_9135b5c5fbc9.slice/crio-013ef21ff71bc6a4f6498e58d7a3c15b025f89a7eed9fa540430b44c8b98b8b9 WatchSource:0}: Error finding container 013ef21ff71bc6a4f6498e58d7a3c15b025f89a7eed9fa540430b44c8b98b8b9: Status 404 returned error can't find the container with id 013ef21ff71bc6a4f6498e58d7a3c15b025f89a7eed9fa540430b44c8b98b8b9 Jul 27 22:24:06.646838 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.646799 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.646952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.646885 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.646952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:06.646930 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:07.174602 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:07.174567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerStarted","Data":"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb"} Jul 27 22:24:07.174810 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:07.174626 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerStarted","Data":"013ef21ff71bc6a4f6498e58d7a3c15b025f89a7eed9fa540430b44c8b98b8b9"} Jul 27 22:24:07.198328 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:07.198283 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podStartSLOduration=1.198267562 podStartE2EDuration="1.198267562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:07.195616501 +0000 UTC m=+1767.576688425" watchObservedRunningTime="2026-07-27 22:24:07.198267562 +0000 UTC m=+1767.579339487" Jul 27 22:24:07.500771 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:07.500678 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:07.505718 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:07.505693 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:08.179297 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.179264 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:08.185955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.185926 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:08.542420 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.542318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:08.638399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.638354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:08.854270 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.854196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:08.990278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:08.990245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:09.502667 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:09.502634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:09.585040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:09.585007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:10.634039 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.634017 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:24:10.739816 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739725 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.739816 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739772 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739821 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nbxq\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739851 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739900 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.739990 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740018 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740073 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:10.740375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740090 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert\") pod \"354455e4-a80a-4889-9c64-12b71a02c375\" (UID: \"354455e4-a80a-4889-9c64-12b71a02c375\") " Jul 27 22:24:10.740375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740321 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:10.740521 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:24:10.740521 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740451 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.740521 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740475 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/354455e4-a80a-4889-9c64-12b71a02c375-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.740521 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740489 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.740771 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740751 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:10.741023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.740997 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data" (OuterVolumeSpecName: "istio-data") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:10.742253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.742230 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq" (OuterVolumeSpecName: "kube-api-access-7nbxq") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "kube-api-access-7nbxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:10.742440 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.742423 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token" (OuterVolumeSpecName: "istio-token") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:10.742733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.742707 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:10.742733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.742706 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "354455e4-a80a-4889-9c64-12b71a02c375" (UID: "354455e4-a80a-4889-9c64-12b71a02c375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:24:10.841117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841079 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.841117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841109 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/354455e4-a80a-4889-9c64-12b71a02c375-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.841117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841121 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.841117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841129 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.841413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841138 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbxq\" (UniqueName: \"kubernetes.io/projected/354455e4-a80a-4889-9c64-12b71a02c375-kube-api-access-7nbxq\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:10.841413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:10.841184 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/354455e4-a80a-4889-9c64-12b71a02c375-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:11.195542 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.195509 2559 generic.go:356] "Generic (PLEG): container finished" podID="354455e4-a80a-4889-9c64-12b71a02c375" containerID="aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054" exitCode=0 Jul 27 22:24:11.195761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.195555 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" Jul 27 22:24:11.195761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.195597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerDied","Data":"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054"} Jul 27 22:24:11.195761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.195667 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6" event={"ID":"354455e4-a80a-4889-9c64-12b71a02c375","Type":"ContainerDied","Data":"3fd1c756adff3e45c7cbb31477312f72d91c77c6aed0c734c7aa1dab22e0616c"} Jul 27 22:24:11.195761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.195701 2559 scope.go:117] "RemoveContainer" containerID="aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054" Jul 27 22:24:11.205618 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.205602 2559 scope.go:117] "RemoveContainer" containerID="6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a" Jul 27 22:24:11.216752 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.216735 2559 scope.go:117] "RemoveContainer" containerID="aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054" Jul 27 22:24:11.217040 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:24:11.217020 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054\": container with ID starting with aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054 not found: ID does not exist" containerID="aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054" Jul 27 22:24:11.217106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.217047 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054"} err="failed to get container status \"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054\": rpc error: code = NotFound desc = could not find container \"aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054\": container with ID starting with aae7f45dbecc745df2a051a4b047306ebde726827904aecab09c5598c289c054 not found: ID does not exist" Jul 27 22:24:11.217106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.217066 2559 scope.go:117] "RemoveContainer" containerID="6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a" Jul 27 22:24:11.217315 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:24:11.217299 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a\": container with ID starting with 6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a not found: ID does not exist" containerID="6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a" Jul 27 22:24:11.217374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.217321 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a"} err="failed to get container status \"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a\": rpc error: code = NotFound desc = could not find container \"6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a\": container with ID starting with 6e22cfd720cb6109feec11eafcc2428c26d28f863f3f8e80775959aa6b938f7a not found: ID does not exist" Jul 27 22:24:11.221851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.221827 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:24:11.225698 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:11.225679 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-294097f5-openshift-default-6f8f9bc49b-76bq6"] Jul 27 22:24:12.084097 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:12.084063 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354455e4-a80a-4889-9c64-12b71a02c375" path="/var/lib/kubelet/pods/354455e4-a80a-4889-9c64-12b71a02c375/volumes" Jul 27 22:24:22.255931 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.255896 2559 generic.go:356] "Generic (PLEG): container finished" podID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerID="1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb" exitCode=0 Jul 27 22:24:22.256518 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.255972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerDied","Data":"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb"} Jul 27 22:24:22.256518 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.256510 2559 scope.go:117] "RemoveContainer" containerID="1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb" Jul 27 22:24:22.257559 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.257532 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:22.257660 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.257596 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:22.257660 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:22.257628 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:23.179745 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:23.179711 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:23.263034 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:23.262997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerStarted","Data":"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb"} Jul 27 22:24:23.501706 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:23.501538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:23.502863 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:23.502837 2559 patch_prober.go:28] interesting pod/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 27 22:24:23.503066 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:23.503042 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 27 22:24:24.501540 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:24.501495 2559 patch_prober.go:28] interesting pod/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 27 22:24:24.501930 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:24.501580 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 27 22:24:25.501051 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:25.501005 2559 patch_prober.go:28] interesting pod/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 27 22:24:25.501264 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:25.501080 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 27 22:24:26.501770 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:26.501724 2559 patch_prober.go:28] interesting pod/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 27 22:24:26.502217 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:26.501800 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 27 22:24:27.506096 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:27.506061 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:27.506495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:27.506440 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:27.507046 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:27.507029 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:33.417117 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417034 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd"] Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417366 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417377 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417391 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417396 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417455 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.417496 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.417464 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="354455e4-a80a-4889-9c64-12b71a02c375" containerName="istio-proxy" Jul 27 22:24:33.420678 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.420658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:33.423373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.423349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:24:33.423473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.423454 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:24:33.430305 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.430279 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd"] Jul 27 22:24:33.552441 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.552395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd\" (UID: \"9b297cc0-3f54-4fcb-bed0-81fbb5c69131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:33.653689 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.653654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd\" (UID: \"9b297cc0-3f54-4fcb-bed0-81fbb5c69131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:33.654273 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.654256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd\" (UID: \"9b297cc0-3f54-4fcb-bed0-81fbb5c69131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:33.732354 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.732268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:33.866685 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:33.866662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd"] Jul 27 22:24:33.868826 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:24:33.868798 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b297cc0_3f54_4fcb_bed0_81fbb5c69131.slice/crio-8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717 WatchSource:0}: Error finding container 8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717: Status 404 returned error can't find the container with id 8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717 Jul 27 22:24:34.318031 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:34.317998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" event={"ID":"9b297cc0-3f54-4fcb-bed0-81fbb5c69131","Type":"ContainerStarted","Data":"6d91e04eba92e7724eb2d666df91b4e3ef2a19bf62da0e90915a7eef7550d917"} Jul 27 22:24:34.318218 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:34.318054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" event={"ID":"9b297cc0-3f54-4fcb-bed0-81fbb5c69131","Type":"ContainerStarted","Data":"8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717"} Jul 27 22:24:34.334948 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:34.334900 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" podStartSLOduration=1.334886515 podStartE2EDuration="1.334886515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:34.333107226 +0000 UTC m=+1794.714179150" watchObservedRunningTime="2026-07-27 22:24:34.334886515 +0000 UTC m=+1794.715958439" Jul 27 22:24:35.324253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:35.324220 2559 generic.go:356] "Generic (PLEG): container finished" podID="9b297cc0-3f54-4fcb-bed0-81fbb5c69131" containerID="6d91e04eba92e7724eb2d666df91b4e3ef2a19bf62da0e90915a7eef7550d917" exitCode=0 Jul 27 22:24:35.324645 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:35.324261 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" event={"ID":"9b297cc0-3f54-4fcb-bed0-81fbb5c69131","Type":"ContainerDied","Data":"6d91e04eba92e7724eb2d666df91b4e3ef2a19bf62da0e90915a7eef7550d917"} Jul 27 22:24:36.457636 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:36.457612 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:36.576357 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:36.576323 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca\") pod \"9b297cc0-3f54-4fcb-bed0-81fbb5c69131\" (UID: \"9b297cc0-3f54-4fcb-bed0-81fbb5c69131\") " Jul 27 22:24:36.576685 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:36.576662 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b297cc0-3f54-4fcb-bed0-81fbb5c69131" (UID: "9b297cc0-3f54-4fcb-bed0-81fbb5c69131"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:24:36.677748 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:36.677714 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b297cc0-3f54-4fcb-bed0-81fbb5c69131-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:37.334826 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:37.334794 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" Jul 27 22:24:37.334826 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:37.334818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd" event={"ID":"9b297cc0-3f54-4fcb-bed0-81fbb5c69131","Type":"ContainerDied","Data":"8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717"} Jul 27 22:24:37.335034 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:37.334868 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7b3f58cc8bad569fadb69cdb5c0e599351c54e1b67a376411f1915f1493717" Jul 27 22:24:40.088889 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:40.088860 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:24:40.092948 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:40.092924 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:24:44.136451 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:44.136411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:44.273484 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:44.273450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:44.389217 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:44.389110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:48.503780 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.503742 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd"] Jul 27 22:24:48.506800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.506773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:48.507258 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.507238 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-858f23ab-6c9qd"] Jul 27 22:24:48.646396 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.646364 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:48.646626 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.646596 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" containerID="cri-o://30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb" gracePeriod=30 Jul 27 22:24:48.670486 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.670450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:48.922201 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:48.922140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:49.431937 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:49.431908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:49.525388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:49.525353 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:49.625559 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:49.625529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:50.083505 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:50.083476 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b297cc0-3f54-4fcb-bed0-81fbb5c69131" path="/var/lib/kubelet/pods/9b297cc0-3f54-4fcb-bed0-81fbb5c69131/volumes" Jul 27 22:24:51.208831 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:51.208773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:53.897827 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:53.897801 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:54.020319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020237 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020313 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwg5p\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020349 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020377 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020410 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020438 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020504 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020544 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs\") pod \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\" (UID: \"2f20593d-2699-4598-8e0e-9135b5c5fbc9\") " Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:54.020598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:54.020999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020775 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:54.020999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020825 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:24:54.020999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020920 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.020999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020941 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.020999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.020956 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.021894 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.021867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:54.022844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.022813 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:24:54.022958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.022813 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:54.022958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.022853 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p" (OuterVolumeSpecName: "kube-api-access-jwg5p") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "kube-api-access-jwg5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:24:54.022958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.022913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f20593d-2699-4598-8e0e-9135b5c5fbc9" (UID: "2f20593d-2699-4598-8e0e-9135b5c5fbc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:24:54.121674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121645 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.121674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121673 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.121854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121684 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg5p\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-kube-api-access-jwg5p\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.121854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121692 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.121854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121700 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.121854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.121708 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f20593d-2699-4598-8e0e-9135b5c5fbc9-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:24:54.424740 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.424704 2559 generic.go:356] "Generic (PLEG): container finished" podID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerID="30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb" exitCode=0 Jul 27 22:24:54.424919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.424755 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" Jul 27 22:24:54.424919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.424784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerDied","Data":"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb"} Jul 27 22:24:54.424919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.424828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls" event={"ID":"2f20593d-2699-4598-8e0e-9135b5c5fbc9","Type":"ContainerDied","Data":"013ef21ff71bc6a4f6498e58d7a3c15b025f89a7eed9fa540430b44c8b98b8b9"} Jul 27 22:24:54.424919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.424852 2559 scope.go:117] "RemoveContainer" containerID="30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb" Jul 27 22:24:54.433933 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.433917 2559 scope.go:117] "RemoveContainer" containerID="1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb" Jul 27 22:24:54.442134 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.442117 2559 scope.go:117] "RemoveContainer" containerID="30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb" Jul 27 22:24:54.442439 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:24:54.442422 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb\": container with ID starting with 30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb not found: ID does not exist" containerID="30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb" Jul 27 22:24:54.442512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.442446 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb"} err="failed to get container status \"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb\": rpc error: code = NotFound desc = could not find container \"30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb\": container with ID starting with 30217e775bf58b14ecfe9c782c1dc428354c2c4cd9252a04b198a1a972ef4bdb not found: ID does not exist" Jul 27 22:24:54.442512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.442464 2559 scope.go:117] "RemoveContainer" containerID="1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb" Jul 27 22:24:54.442749 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:24:54.442727 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb\": container with ID starting with 1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb not found: ID does not exist" containerID="1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb" Jul 27 22:24:54.442791 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.442757 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb"} err="failed to get container status \"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb\": rpc error: code = NotFound desc = could not find container \"1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb\": container with ID starting with 1e2f760747369cde9033414f09b77c5fc32f8843bf7f4e62fe90058914a9cdfb not found: ID does not exist" Jul 27 22:24:54.443177 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.443140 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:54.448641 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:54.448621 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-858f23ab-openshift-default-67967b9676-9l7ls"] Jul 27 22:24:56.083663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:56.083628 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" path="/var/lib/kubelet/pods/2f20593d-2699-4598-8e0e-9135b5c5fbc9/volumes" Jul 27 22:24:57.860067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:57.860029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:57.963317 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:57.963284 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:58.066741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:58.066685 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:58.166805 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:58.166699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:58.576432 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:24:58.576326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:24.578304 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:24.578263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:31.097259 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:31.097214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:31.205790 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:31.205755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:31.312847 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:31.312808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:44.476047 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:44.476010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:47.223553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:47.223513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:52.961712 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:52.961673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:53.052373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:53.052331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:53.151945 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:25:53.151908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:03.671826 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:03.671729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:04.492935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:04.492901 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:11.646931 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:11.646885 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:11.759607 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:11.759566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:11.847936 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:11.847896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:11.953550 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:11.953453 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:12.757520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:12.757488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:12.858941 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:12.858882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:12.955714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:12.955657 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:14.552042 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:14.551990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:20.240642 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:20.240599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:20.307370 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:20.307333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:20.430710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:20.430677 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:20.531228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:20.531113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:20.904421 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:20.904393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:26.978746 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:26.978684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:27.082075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.082037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:27.227219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.227180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:27.525927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.525896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:26:27.526220 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526206 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:27.526291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526221 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:27.526291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526236 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b297cc0-3f54-4fcb-bed0-81fbb5c69131" containerName="revoke-keys" Jul 27 22:26:27.526291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526242 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b297cc0-3f54-4fcb-bed0-81fbb5c69131" containerName="revoke-keys" Jul 27 22:26:27.526403 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526292 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:27.526403 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526300 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b297cc0-3f54-4fcb-bed0-81fbb5c69131" containerName="revoke-keys" Jul 27 22:26:27.526403 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.526305 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:27.529221 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.529206 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.532316 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.532286 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-92c78b5c-openshift-default-dockercfg-q67q7\"" Jul 27 22:26:27.557173 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.557119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:26:27.558593 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.558556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:27.620614 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.620798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.620798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.620932 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrk4\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.620932 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.620932 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.621083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.621083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.620987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.621083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.621018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722395 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722395 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrk4\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.722981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.722860 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.723192 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.723020 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.723192 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.723081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.723192 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.723175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.723355 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.723212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.725784 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.725753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.725895 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.725879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.731939 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.731906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrk4\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.732069 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.731990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token\") pod \"e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.839926 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.839581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:27.944135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:27.944098 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:28.007280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.007253 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:26:28.010369 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:26:28.010337 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea16c204_ec71_4fd4_bbc7_ed98c0b1a40d.slice/crio-bb8b3dab0ad860bd9531c29b057fab27621f83522ebc63cc0ac598b06cbb8a79 WatchSource:0}: Error finding container bb8b3dab0ad860bd9531c29b057fab27621f83522ebc63cc0ac598b06cbb8a79: Status 404 returned error can't find the container with id bb8b3dab0ad860bd9531c29b057fab27621f83522ebc63cc0ac598b06cbb8a79 Jul 27 22:26:28.012601 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.012559 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:28.012687 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.012653 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:28.012729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.012699 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:28.911088 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.911046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerStarted","Data":"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d"} Jul 27 22:26:28.911278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.911096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerStarted","Data":"bb8b3dab0ad860bd9531c29b057fab27621f83522ebc63cc0ac598b06cbb8a79"} Jul 27 22:26:28.947487 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:28.947429 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podStartSLOduration=1.9474132119999998 podStartE2EDuration="1.947413212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:26:28.945899721 +0000 UTC m=+1909.326971656" watchObservedRunningTime="2026-07-27 22:26:28.947413212 +0000 UTC m=+1909.328485136" Jul 27 22:26:29.840497 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:29.840462 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:29.845472 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:29.845449 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:29.915647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:29.915618 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:29.916628 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:29.916608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:30.974820 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:30.974775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.079305 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:31.079271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.233801 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:31.233715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.389960 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:31.389928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.751734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:31.751693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.851359 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:31.851308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:35.466176 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:35.466116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:36.166591 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:36.166558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:36.264257 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:36.264220 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:36.609538 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:36.609510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:42.305922 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.305888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:42.369593 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.369558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:42.668589 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.668559 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d"] Jul 27 22:26:42.668873 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.668861 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:42.668920 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.668874 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f20593d-2699-4598-8e0e-9135b5c5fbc9" containerName="istio-proxy" Jul 27 22:26:42.671665 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.671647 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:42.674056 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.674032 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:26:42.674227 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.674072 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:26:42.690568 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.690526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d"] Jul 27 22:26:42.762436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.762370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d\" (UID: \"e2872ac8-d415-49b6-a68e-7aa8f74a4191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:42.863507 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.863457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d\" (UID: \"e2872ac8-d415-49b6-a68e-7aa8f74a4191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:42.864118 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:42.864096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d\" (UID: \"e2872ac8-d415-49b6-a68e-7aa8f74a4191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:43.018965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:43.018922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:43.160738 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:43.160712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d"] Jul 27 22:26:43.162442 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:26:43.162413 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2872ac8_d415_49b6_a68e_7aa8f74a4191.slice/crio-84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7 WatchSource:0}: Error finding container 84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7: Status 404 returned error can't find the container with id 84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7 Jul 27 22:26:43.984787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:43.984751 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2872ac8-d415-49b6-a68e-7aa8f74a4191" containerID="0f9e7c219229f4e76575e265d131c0498f5a67673fd879bfcff970dc3a562d81" exitCode=0 Jul 27 22:26:43.985206 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:43.984844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" event={"ID":"e2872ac8-d415-49b6-a68e-7aa8f74a4191","Type":"ContainerDied","Data":"0f9e7c219229f4e76575e265d131c0498f5a67673fd879bfcff970dc3a562d81"} Jul 27 22:26:43.985206 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:43.984912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" event={"ID":"e2872ac8-d415-49b6-a68e-7aa8f74a4191","Type":"ContainerStarted","Data":"84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7"} Jul 27 22:26:45.122452 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.122425 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:45.182554 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.182521 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca\") pod \"e2872ac8-d415-49b6-a68e-7aa8f74a4191\" (UID: \"e2872ac8-d415-49b6-a68e-7aa8f74a4191\") " Jul 27 22:26:45.182924 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.182897 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2872ac8-d415-49b6-a68e-7aa8f74a4191" (UID: "e2872ac8-d415-49b6-a68e-7aa8f74a4191"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:26:45.283512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.283426 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2872ac8-d415-49b6-a68e-7aa8f74a4191-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:26:45.443412 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.443375 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:51732->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 27 22:26:45.443599 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.443457 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:51732->10.133.0.43:15021: read: connection reset by peer" Jul 27 22:26:45.995851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.995822 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerID="f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d" exitCode=0 Jul 27 22:26:45.996063 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.995901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerDied","Data":"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d"} Jul 27 22:26:45.996375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.996356 2559 scope.go:117] "RemoveContainer" containerID="f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d" Jul 27 22:26:45.997174 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997138 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" Jul 27 22:26:45.997253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d" event={"ID":"e2872ac8-d415-49b6-a68e-7aa8f74a4191","Type":"ContainerDied","Data":"84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7"} Jul 27 22:26:45.997253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997227 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d46c4eedc8e220e3079c2f78713744e8f3eb1168f0b5a13c87813ebbfee7b7" Jul 27 22:26:45.997389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997362 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:45.997452 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997437 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:45.997511 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:45.997483 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:26:47.003512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:47.003481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerStarted","Data":"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d"} Jul 27 22:26:47.841204 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:47.841140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:47.841398 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:47.841343 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:26:47.841451 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:47.841393 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:26:48.841374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:48.841336 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:26:48.841803 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:48.841394 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:26:49.841449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:49.841415 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:26:49.841859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:49.841471 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:26:50.840856 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:50.840818 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:26:50.841040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:50.840886 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:26:51.844343 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:51.844310 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:51.844713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:51.844605 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:51.845453 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:51.845436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:26:53.032046 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:53.032006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:53.101748 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:53.101703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:53.161817 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:53.161768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.742261 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:57.742202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.746558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:57.746527 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d"] Jul 27 22:26:57.748749 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:57.748724 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-92c78b5c-gsg5d"] Jul 27 22:26:58.083697 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:58.083605 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2872ac8-d415-49b6-a68e-7aa8f74a4191" path="/var/lib/kubelet/pods/e2872ac8-d415-49b6-a68e-7aa8f74a4191/volumes" Jul 27 22:26:58.144305 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:26:58.144272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:04.329663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.329625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:04.430605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.430569 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:04.587102 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.587017 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:27:04.587298 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.587262 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" containerID="cri-o://85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d" gracePeriod=30 Jul 27 22:27:04.614902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.614867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:04.850508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:04.850419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:05.010443 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:05.010403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:05.106227 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:05.106114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:05.223402 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:05.223371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:06.846842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:06.846807 2559 patch_prober.go:28] interesting pod/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:27:06.847256 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:06.846865 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:27:09.834787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.834765 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:27:09.908222 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908184 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908276 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhrk4\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908299 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908337 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908395 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908418 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908457 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908512 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy\") pod \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\" (UID: \"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d\") " Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908505 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908602 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908753 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:09.908799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.908771 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:27:09.909334 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.909280 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:09.910652 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.910633 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:09.910801 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.910786 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4" (OuterVolumeSpecName: "kube-api-access-rhrk4") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "kube-api-access-rhrk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:27:09.911123 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.911110 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:27:09.911193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:09.911167 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" (UID: "ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:27:10.010006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.009965 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.009998 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrk4\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-kube-api-access-rhrk4\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010011 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010019 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010028 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010036 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010043 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010050 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.010260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.010057 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:27:10.122211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.122176 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerID="85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d" exitCode=0 Jul 27 22:27:10.122381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.122233 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" Jul 27 22:27:10.122381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.122258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerDied","Data":"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d"} Jul 27 22:27:10.122381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.122309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg" event={"ID":"ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d","Type":"ContainerDied","Data":"bb8b3dab0ad860bd9531c29b057fab27621f83522ebc63cc0ac598b06cbb8a79"} Jul 27 22:27:10.122381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.122332 2559 scope.go:117] "RemoveContainer" containerID="85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d" Jul 27 22:27:10.131299 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.131285 2559 scope.go:117] "RemoveContainer" containerID="f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d" Jul 27 22:27:10.139391 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.139374 2559 scope.go:117] "RemoveContainer" containerID="85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d" Jul 27 22:27:10.139681 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:27:10.139645 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d\": container with ID starting with 85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d not found: ID does not exist" containerID="85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d" Jul 27 22:27:10.139726 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.139691 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d"} err="failed to get container status \"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d\": rpc error: code = NotFound desc = could not find container \"85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d\": container with ID starting with 85dbd7df91fe89cbaaee8e3c2ca3d8273f10d824198e6c381838cdfe6e55d31d not found: ID does not exist" Jul 27 22:27:10.139726 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.139708 2559 scope.go:117] "RemoveContainer" containerID="f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d" Jul 27 22:27:10.139971 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:27:10.139945 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d\": container with ID starting with f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d not found: ID does not exist" containerID="f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d" Jul 27 22:27:10.140017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.139983 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d"} err="failed to get container status \"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d\": rpc error: code = NotFound desc = could not find container \"f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d\": container with ID starting with f284694b45f7b7da08175210ffb7852ede90c17d4af7acfed5fa29665cab1a9d not found: ID does not exist" Jul 27 22:27:10.140869 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.140853 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:27:10.148475 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:10.148452 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-92c78b5c-openshift-default-5fd7599bd8-59nwg"] Jul 27 22:27:11.290996 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:11.290963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:11.409286 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:11.409249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:11.531563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:11.531534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:11.672748 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:11.672705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:12.084052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:12.083970 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" path="/var/lib/kubelet/pods/ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d/volumes" Jul 27 22:27:17.856605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:17.856571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:17.929291 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:17.929255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:24.673843 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.673802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:24.969504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969422 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:27:24.969766 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969749 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969777 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969792 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2872ac8-d415-49b6-a68e-7aa8f74a4191" containerName="revoke-keys" Jul 27 22:27:24.969819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969797 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2872ac8-d415-49b6-a68e-7aa8f74a4191" containerName="revoke-keys" Jul 27 22:27:24.969986 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969862 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969986 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969873 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea16c204-ec71-4fd4-bbc7-ed98c0b1a40d" containerName="istio-proxy" Jul 27 22:27:24.969986 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.969883 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2872ac8-d415-49b6-a68e-7aa8f74a4191" containerName="revoke-keys" Jul 27 22:27:24.973116 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.973098 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:24.975502 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.975482 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d2650623-openshift-default-dockercfg-t5gkn\"" Jul 27 22:27:24.987072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:24.987048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:27:25.002331 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.002305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:25.033539 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.033950 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.033887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8j9\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135038 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8j9\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135836 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.135892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.136047 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.135964 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.137591 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.137568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.137869 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.137849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.143881 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.143858 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.144112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.144089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8j9\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9\") pod \"e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.267526 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.267440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:25.282386 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.282364 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:25.421223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.415370 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:27:25.421223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.420310 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:25.421223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.420444 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:25.421223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:25.420549 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:26.201137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:26.201108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerStarted","Data":"8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf"} Jul 27 22:27:26.201533 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:26.201173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerStarted","Data":"413775e047c390b75d88ab28671477ceb5f6015ffdd6d90a12853146db9b5797"} Jul 27 22:27:26.221882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:26.221836 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podStartSLOduration=2.221820613 podStartE2EDuration="2.221820613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:27:26.220367156 +0000 UTC m=+1966.601439080" watchObservedRunningTime="2026-07-27 22:27:26.221820613 +0000 UTC m=+1966.602892537" Jul 27 22:27:26.283583 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:26.283549 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:26.288211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:26.288185 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:27.206428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:27.206390 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:27.207414 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:27.207396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:28.131405 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.131371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:28.238402 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.238367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:28.384473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.384378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:28.723401 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.723315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:28.888200 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.888143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:28.991224 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:28.991108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:32.621709 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:32.621617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:41.607463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:41.607419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:42.737579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:42.736926 2559 patch_prober.go:28] interesting pod/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:45116->10.133.0.45:15021: read: connection reset by peer" start-of-body= Jul 27 22:27:42.737579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:42.736986 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:45116->10.133.0.45:15021: read: connection reset by peer" Jul 27 22:27:43.285413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.285381 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerID="8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf" exitCode=0 Jul 27 22:27:43.285597 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.285448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerDied","Data":"8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf"} Jul 27 22:27:43.285819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.285806 2559 scope.go:117] "RemoveContainer" containerID="8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf" Jul 27 22:27:43.286797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.286767 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:43.286954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.286832 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:43.286954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:43.286861 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:27:44.299370 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:44.299330 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerStarted","Data":"0c38f0fafb91f482009318913ae28fe205415032ba662e4be7d2b9b98d5f04da"} Jul 27 22:27:45.283543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:45.283498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:45.283778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:45.283643 2559 patch_prober.go:28] interesting pod/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:27:45.283778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:45.283692 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:27:46.283845 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:46.283804 2559 patch_prober.go:28] interesting pod/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:27:46.284483 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:46.284446 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:27:47.283903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:47.283418 2559 patch_prober.go:28] interesting pod/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 27 22:27:47.283903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:47.283496 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 27 22:27:48.286682 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:48.286650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:48.287093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:48.287003 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:27:48.287585 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:27:48.287568 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:28:07.095443 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.095398 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8"] Jul 27 22:28:07.099208 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.099187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:07.101535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.101514 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:28:07.101648 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.101545 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:28:07.105901 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.105880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8"] Jul 27 22:28:07.194953 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.194911 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2650623-782n8\" (UID: \"9876a87d-f41a-47a5-9b73-bb6699a58616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:07.295504 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.295476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2650623-782n8\" (UID: \"9876a87d-f41a-47a5-9b73-bb6699a58616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:07.296092 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.296074 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2650623-782n8\" (UID: \"9876a87d-f41a-47a5-9b73-bb6699a58616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:07.410728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.410685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:07.539896 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:07.539871 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8"] Jul 27 22:28:07.541665 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:28:07.541638 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9876a87d_f41a_47a5_9b73_bb6699a58616.slice/crio-a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e WatchSource:0}: Error finding container a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e: Status 404 returned error can't find the container with id a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e Jul 27 22:28:08.424261 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:08.424226 2559 generic.go:356] "Generic (PLEG): container finished" podID="9876a87d-f41a-47a5-9b73-bb6699a58616" containerID="c691c796c65045812ed8969adfa175908e9dd571e20dd15ab26e4498d4a16043" exitCode=0 Jul 27 22:28:08.424635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:08.424297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" event={"ID":"9876a87d-f41a-47a5-9b73-bb6699a58616","Type":"ContainerDied","Data":"c691c796c65045812ed8969adfa175908e9dd571e20dd15ab26e4498d4a16043"} Jul 27 22:28:08.424635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:08.424335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" event={"ID":"9876a87d-f41a-47a5-9b73-bb6699a58616","Type":"ContainerStarted","Data":"a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e"} Jul 27 22:28:09.554903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:09.554878 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:09.614112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:09.614082 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca\") pod \"9876a87d-f41a-47a5-9b73-bb6699a58616\" (UID: \"9876a87d-f41a-47a5-9b73-bb6699a58616\") " Jul 27 22:28:09.614444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:09.614422 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9876a87d-f41a-47a5-9b73-bb6699a58616" (UID: "9876a87d-f41a-47a5-9b73-bb6699a58616"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:28:09.715103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:09.715015 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9876a87d-f41a-47a5-9b73-bb6699a58616-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.434569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:10.434539 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" Jul 27 22:28:10.434764 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:10.434577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8" event={"ID":"9876a87d-f41a-47a5-9b73-bb6699a58616","Type":"ContainerDied","Data":"a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e"} Jul 27 22:28:10.434764 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:10.434632 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e60548ca92a7fa3d4a9a0afbe8681f348b6042095be31c598bfb4aa912499e" Jul 27 22:28:22.786676 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:22.786643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:23.995014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:23.994980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:27.182579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:27.182540 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:27.187529 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:27.187505 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8"] Jul 27 22:28:27.190418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:27.190397 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2650623-782n8"] Jul 27 22:28:28.083802 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:28.083768 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9876a87d-f41a-47a5-9b73-bb6699a58616" path="/var/lib/kubelet/pods/9876a87d-f41a-47a5-9b73-bb6699a58616/volumes" Jul 27 22:28:30.422384 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:30.422349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:35.440489 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:28:35.440457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:12.148240 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:12.148120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:12.445586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:12.445491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:12.784065 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:12.783981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:18.299212 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.299179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:18.400056 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.400009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:18.561131 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.561052 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:29:18.561339 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.561313 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" containerID="cri-o://0c38f0fafb91f482009318913ae28fe205415032ba662e4be7d2b9b98d5f04da" gracePeriod=30 Jul 27 22:29:18.595468 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.595427 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:18.824837 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:18.824748 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:19.851635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:19.851566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:19.949346 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:19.949313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:20.135917 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:20.135882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:20.832512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:20.832476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:20.929613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:20.929574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:21.029200 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:21.029138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:22.379549 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:22.379513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:23.794969 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.794935 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerID="0c38f0fafb91f482009318913ae28fe205415032ba662e4be7d2b9b98d5f04da" exitCode=0 Jul 27 22:29:23.795414 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.794998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerDied","Data":"0c38f0fafb91f482009318913ae28fe205415032ba662e4be7d2b9b98d5f04da"} Jul 27 22:29:23.795414 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.795070 2559 scope.go:117] "RemoveContainer" containerID="8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf" Jul 27 22:29:23.818410 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:29:23.818370 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf\": container with ID starting with 8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf not found: ID does not exist" containerID="8ac3709ec253e62bd12597ad37e89cf0115a41f9bbaf0bd78059dbca463f7adf" Jul 27 22:29:23.821810 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.821788 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:29:23.948219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948277 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948299 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948348 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948371 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl8j9\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948682 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948613 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:29:23.948682 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:29:23.948793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948694 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:29:23.948793 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948793 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.948897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.948825 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert\") pod \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\" (UID: \"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c\") " Jul 27 22:29:23.949118 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.949093 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:23.949248 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.949122 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:23.949248 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.949137 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:23.949449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.949427 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:29:23.949735 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.949705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:29:23.951242 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.951202 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9" (OuterVolumeSpecName: "kube-api-access-vl8j9") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "kube-api-access-vl8j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:29:23.951350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.951311 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:29:23.951418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.951347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:29:23.952048 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:23.952026 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" (UID: "a2b1b208-4e24-4b30-8c0d-40c5fb72f47c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:29:24.049686 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049647 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.049686 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049674 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.049686 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049686 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8j9\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-kube-api-access-vl8j9\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.049686 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049694 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.049946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049705 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.049946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.049713 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:29:24.801175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.801129 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" Jul 27 22:29:24.801627 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.801127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr" event={"ID":"a2b1b208-4e24-4b30-8c0d-40c5fb72f47c","Type":"ContainerDied","Data":"413775e047c390b75d88ab28671477ceb5f6015ffdd6d90a12853146db9b5797"} Jul 27 22:29:24.801627 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.801268 2559 scope.go:117] "RemoveContainer" containerID="0c38f0fafb91f482009318913ae28fe205415032ba662e4be7d2b9b98d5f04da" Jul 27 22:29:24.819097 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.819071 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:29:24.824927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:24.824904 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d2650623-openshift-default-6d9749766d-7jcjr"] Jul 27 22:29:26.083807 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:26.083775 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" path="/var/lib/kubelet/pods/a2b1b208-4e24-4b30-8c0d-40c5fb72f47c/volumes" Jul 27 22:29:28.011508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:28.011469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:28.082144 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:28.082112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:28.659214 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:28.659181 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:35.306130 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:35.306090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:35.417607 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:35.417562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:35.540484 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:35.540436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:40.126567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:40.126532 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:29:40.131815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:40.131788 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:29:40.180995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:40.180961 2559 scope.go:117] "RemoveContainer" containerID="60d57fdb77bee243cdadb8c388e4f48f568396be0279c697a195bcc9984273ed" Jul 27 22:29:46.035055 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:46.035020 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:46.380554 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:29:46.380503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:00.142075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142041 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd"] Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142336 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142347 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142355 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9876a87d-f41a-47a5-9b73-bb6699a58616" containerName="revoke-keys" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142360 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9876a87d-f41a-47a5-9b73-bb6699a58616" containerName="revoke-keys" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142369 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142375 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142432 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142440 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9876a87d-f41a-47a5-9b73-bb6699a58616" containerName="revoke-keys" Jul 27 22:30:00.142548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.142448 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b1b208-4e24-4b30-8c0d-40c5fb72f47c" containerName="istio-proxy" Jul 27 22:30:00.145364 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.145348 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" Jul 27 22:30:00.147535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.147512 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:30:00.153571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.153547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd"] Jul 27 22:30:00.155995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.155974 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" Jul 27 22:30:00.286788 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.286765 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd"] Jul 27 22:30:00.288353 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:30:00.288326 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd693fa0_3d5c_4cd3_896c_a66a8fab5118.slice/crio-afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52 WatchSource:0}: Error finding container afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52: Status 404 returned error can't find the container with id afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52 Jul 27 22:30:00.978278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.978241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" event={"ID":"bd693fa0-3d5c-4cd3-896c-a66a8fab5118","Type":"ContainerStarted","Data":"41127c35d8d1d89188f2a03cf205f5c17a288fc1b84027732a8fd2b07d6cd27f"} Jul 27 22:30:00.978488 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.978291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" event={"ID":"bd693fa0-3d5c-4cd3-896c-a66a8fab5118","Type":"ContainerStarted","Data":"afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52"} Jul 27 22:30:00.994163 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:00.994098 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" podStartSLOduration=0.994084129 podStartE2EDuration="994.084129ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:00.993398222 +0000 UTC m=+2121.374470146" watchObservedRunningTime="2026-07-27 22:30:00.994084129 +0000 UTC m=+2121.375156052" Jul 27 22:30:01.984245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:01.984210 2559 generic.go:356] "Generic (PLEG): container finished" podID="bd693fa0-3d5c-4cd3-896c-a66a8fab5118" containerID="41127c35d8d1d89188f2a03cf205f5c17a288fc1b84027732a8fd2b07d6cd27f" exitCode=0 Jul 27 22:30:01.984622 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:01.984265 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" event={"ID":"bd693fa0-3d5c-4cd3-896c-a66a8fab5118","Type":"ContainerDied","Data":"41127c35d8d1d89188f2a03cf205f5c17a288fc1b84027732a8fd2b07d6cd27f"} Jul 27 22:30:03.110879 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:03.110854 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" Jul 27 22:30:03.994495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:03.994465 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" Jul 27 22:30:03.994639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:03.994497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-q4brd" event={"ID":"bd693fa0-3d5c-4cd3-896c-a66a8fab5118","Type":"ContainerDied","Data":"afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52"} Jul 27 22:30:03.994639 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:03.994564 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd84b49db638e70ce3eacbb358470c0370f3da6882b221cb933a49f3df64f52" Jul 27 22:30:07.372259 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:07.372219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:13.590531 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:13.590465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:13.686721 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:13.686685 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:13.954554 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:13.954519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:14.254393 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.254312 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:30:14.254649 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.254634 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd693fa0-3d5c-4cd3-896c-a66a8fab5118" containerName="cleanup" Jul 27 22:30:14.254692 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.254652 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd693fa0-3d5c-4cd3-896c-a66a8fab5118" containerName="cleanup" Jul 27 22:30:14.254731 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.254712 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd693fa0-3d5c-4cd3-896c-a66a8fab5118" containerName="cleanup" Jul 27 22:30:14.257514 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.257493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.260104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.260081 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5bbdd-openshift-default-dockercfg-sbdwc\"" Jul 27 22:30:14.276660 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.276622 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:30:14.284094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.284068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:14.381098 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381272 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25s2\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381272 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381272 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.381419 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.381414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.389071 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.389038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:14.482724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25s2\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482903 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.482965 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.482938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.483389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.483320 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.483389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.483377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.483501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.483478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.483734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.483525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.483734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.483697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.485228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.485209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.485449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.485433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.490457 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.490429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.490573 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.490550 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25s2\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2\") pod \"e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.568682 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.568582 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:14.709321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.709290 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:30:14.711125 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:30:14.711097 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41583211_0aac_4a7d_bdd9_bc446331e6cf.slice/crio-5201d0a3758d33e17736f715497e3b9d32470e1a893ba8eb0653a313e5dfc39f WatchSource:0}: Error finding container 5201d0a3758d33e17736f715497e3b9d32470e1a893ba8eb0653a313e5dfc39f: Status 404 returned error can't find the container with id 5201d0a3758d33e17736f715497e3b9d32470e1a893ba8eb0653a313e5dfc39f Jul 27 22:30:14.713252 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.713218 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:14.713332 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.713284 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:14.713332 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:14.713313 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:15.051889 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:15.051855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerStarted","Data":"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf"} Jul 27 22:30:15.052101 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:15.051919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerStarted","Data":"5201d0a3758d33e17736f715497e3b9d32470e1a893ba8eb0653a313e5dfc39f"} Jul 27 22:30:15.568912 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:15.568879 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:15.573500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:15.573479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:15.598247 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:15.598188 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podStartSLOduration=1.598165611 podStartE2EDuration="1.598165611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:15.079385248 +0000 UTC m=+2135.460457182" watchObservedRunningTime="2026-07-27 22:30:15.598165611 +0000 UTC m=+2135.979237528" Jul 27 22:30:16.057001 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:16.056972 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:16.057887 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:16.057866 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:17.041810 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.041773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:17.127430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.127375 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:17.306372 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.306273 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:17.420517 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.420478 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:17.839448 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.839391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:17.941387 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:17.941356 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:21.528713 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:21.528675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:21.877594 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:21.877559 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:22.189440 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.189354 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:30:22.193165 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.193127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.196068 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.196042 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2aa15-openshift-default-dockercfg-dqxxt\"" Jul 27 22:30:22.204958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.204932 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:30:22.220451 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.220425 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:22.348254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2zw\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348489 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.348637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.348519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449622 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449622 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449871 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.449903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450110 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450110 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.449939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2zw\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450110 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.450044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.450141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.450303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.450518 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.450493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.452449 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.452425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.452642 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.452595 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.461072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.461052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.461072 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.461067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2zw\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw\") pod \"e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.493388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.493347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:22.504522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.504487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:22.649418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.649389 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:30:22.651754 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:30:22.651726 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce438ec2_7176_4677_a6a7_3103522bb4a4.slice/crio-292b36be9420973be5a83e5570d4279bf17402fd0b59c0c5906a318e4190e751 WatchSource:0}: Error finding container 292b36be9420973be5a83e5570d4279bf17402fd0b59c0c5906a318e4190e751: Status 404 returned error can't find the container with id 292b36be9420973be5a83e5570d4279bf17402fd0b59c0c5906a318e4190e751 Jul 27 22:30:22.653865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.653832 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:22.653945 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.653900 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:22.653945 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:22.653934 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:23.104339 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:23.104253 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerStarted","Data":"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac"} Jul 27 22:30:23.104339 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:23.104311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerStarted","Data":"292b36be9420973be5a83e5570d4279bf17402fd0b59c0c5906a318e4190e751"} Jul 27 22:30:23.123391 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:23.123301 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podStartSLOduration=1.123278716 podStartE2EDuration="1.123278716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:23.123218031 +0000 UTC m=+2143.504289956" watchObservedRunningTime="2026-07-27 22:30:23.123278716 +0000 UTC m=+2143.504350641" Jul 27 22:30:23.505661 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:23.505616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:23.510554 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:23.510526 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:24.109466 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:24.109433 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:24.110623 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:24.110602 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:25.056902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.056868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.141049 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.141015 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.212775 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.212740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.346015 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.345917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.809532 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.809463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:25.905485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:25.905447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:29.529408 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:29.529321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:31.976465 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:31.975644 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:57760->10.133.0.48:15021: read: connection reset by peer" start-of-body= Jul 27 22:30:31.976465 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:31.975705 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:57760->10.133.0.48:15021: read: connection reset by peer" Jul 27 22:30:32.151797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.151706 2559 generic.go:356] "Generic (PLEG): container finished" podID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerID="3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf" exitCode=0 Jul 27 22:30:32.151974 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.151786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerDied","Data":"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf"} Jul 27 22:30:32.152254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.152237 2559 scope.go:117] "RemoveContainer" containerID="3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf" Jul 27 22:30:32.153303 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.153273 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:32.153397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.153353 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:32.153459 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:32.153403 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:33.158875 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:33.158846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerStarted","Data":"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc"} Jul 27 22:30:33.569749 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:33.569655 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:33.570025 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:33.569991 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:30:33.570075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:33.570052 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:30:34.569567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:34.569530 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:30:34.570059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:34.569591 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:30:35.359900 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:35.359850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:30:35.569311 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:35.569271 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:30:35.569512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:35.569334 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:30:36.569631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:36.569590 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 27 22:30:36.570010 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:36.569662 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 27 22:30:37.573036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:37.573007 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:37.573433 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:37.573374 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:37.574112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:37.574094 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:30:39.110829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.110796 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 27 22:30:39.111257 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.110849 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 27 22:30:39.190630 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.190555 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerID="fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac" exitCode=0 Jul 27 22:30:39.190770 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.190629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerDied","Data":"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac"} Jul 27 22:30:39.191207 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.191190 2559 scope.go:117] "RemoveContainer" containerID="fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac" Jul 27 22:30:39.192231 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.192204 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:39.192309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.192272 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:39.192346 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:39.192306 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:30:40.203347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.203306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerStarted","Data":"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8"} Jul 27 22:30:40.205877 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.205850 2559 scope.go:117] "RemoveContainer" containerID="5f6061255c7173aa83a1f46da171284a372ca9e5b2c07a3721acefc3030ca8db" Jul 27 22:30:40.217271 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.217252 2559 scope.go:117] "RemoveContainer" containerID="6d91e04eba92e7724eb2d666df91b4e3ef2a19bf62da0e90915a7eef7550d917" Jul 27 22:30:40.505198 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.505078 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:40.505426 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.505403 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 27 22:30:40.505474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:40.505457 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 27 22:30:41.505588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:41.505545 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 27 22:30:41.505989 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:41.505604 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 27 22:30:42.505014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:42.504976 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 27 22:30:42.505251 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:42.505039 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 27 22:30:43.505754 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:43.505712 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 27 22:30:43.506282 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:43.505775 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 27 22:30:44.508931 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:44.508900 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:44.509400 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:44.509175 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:30:44.509849 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:30:44.509832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:31:02.040912 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.040882 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:31:02.044219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.044202 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.046759 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.046735 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2aa15\"/\"openshift-service-ca.crt\"" Jul 27 22:31:02.047562 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.047542 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2aa15\"/\"default-dockercfg-sltsx\"" Jul 27 22:31:02.047671 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.047560 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2aa15\"/\"test-model-b2aa15-kserve-self-signed-certs\"" Jul 27 22:31:02.047671 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.047542 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2aa15\"/\"kube-root-ca.crt\"" Jul 27 22:31:02.057286 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.057266 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:31:02.076824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.076798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:02.118903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.118854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.119067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.118913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.119067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.118938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.119067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.118955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8nvq\" (UniqueName: \"kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.119067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.118979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.119067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.119011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8nvq\" (UniqueName: \"kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220845 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.220953 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.220871 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.221245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.221087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.223954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.223926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.224206 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.224181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.229659 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.229628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8nvq\" (UniqueName: \"kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq\") pod \"test-model-b2aa15-kserve-bc9bd6dbf-5gqqp\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.353334 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.353236 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:02.488891 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:02.488865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:31:02.491028 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:31:02.490997 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2049156d_b9ea_4df4_a3c5_9ad0737af5c6.slice/crio-e911e5e7ba591fe15831afe0578fdf8f9ecc5eade9e6f3a36e6721f59583b875 WatchSource:0}: Error finding container e911e5e7ba591fe15831afe0578fdf8f9ecc5eade9e6f3a36e6721f59583b875: Status 404 returned error can't find the container with id e911e5e7ba591fe15831afe0578fdf8f9ecc5eade9e6f3a36e6721f59583b875 Jul 27 22:31:03.320243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:03.320209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" event={"ID":"2049156d-b9ea-4df4-a3c5-9ad0737af5c6","Type":"ContainerStarted","Data":"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b"} Jul 27 22:31:03.320751 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:03.320252 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" event={"ID":"2049156d-b9ea-4df4-a3c5-9ad0737af5c6","Type":"ContainerStarted","Data":"e911e5e7ba591fe15831afe0578fdf8f9ecc5eade9e6f3a36e6721f59583b875"} Jul 27 22:31:03.340631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:03.340584 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" podStartSLOduration=1.340570986 podStartE2EDuration="1.340570986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:31:03.337749097 +0000 UTC m=+2183.718821021" watchObservedRunningTime="2026-07-27 22:31:03.340570986 +0000 UTC m=+2183.721642910" Jul 27 22:31:11.051091 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:11.051055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:11.152267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:11.152236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:12.353616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:12.353577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:12.353616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:12.353625 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:12.366450 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:12.366425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:12.378421 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:12.378394 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:31:12.835722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:12.835690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:43.880688 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:43.880644 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:44.741635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.741602 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:31:44.763804 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.763776 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:44.886374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.886341 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk"] Jul 27 22:31:44.889512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.889496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:44.891830 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.891804 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:31:44.891961 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.891810 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:31:44.897677 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.897652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk"] Jul 27 22:31:44.999045 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:44.998954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk\" (UID: \"b26a26f7-8e26-419d-abb1-2e793d5696d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:45.099766 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.099734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk\" (UID: \"b26a26f7-8e26-419d-abb1-2e793d5696d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:45.100381 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.100362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk\" (UID: \"b26a26f7-8e26-419d-abb1-2e793d5696d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:45.200870 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.200833 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:45.335761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.335730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk"] Jul 27 22:31:45.338987 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:31:45.338948 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26a26f7_8e26_419d_abb1_2e793d5696d3.slice/crio-5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf WatchSource:0}: Error finding container 5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf: Status 404 returned error can't find the container with id 5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf Jul 27 22:31:45.532199 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.532041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" event={"ID":"b26a26f7-8e26-419d-abb1-2e793d5696d3","Type":"ContainerStarted","Data":"83a87e4d76daf76a1b77e0df38a98524fd9c2fd6222066680940c077da5e0375"} Jul 27 22:31:45.532199 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.532136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" event={"ID":"b26a26f7-8e26-419d-abb1-2e793d5696d3","Type":"ContainerStarted","Data":"5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf"} Jul 27 22:31:45.549075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:45.549019 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" podStartSLOduration=1.54900277 podStartE2EDuration="1.54900277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:31:45.546467927 +0000 UTC m=+2225.927539851" watchObservedRunningTime="2026-07-27 22:31:45.54900277 +0000 UTC m=+2225.930074694" Jul 27 22:31:46.538873 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:46.538837 2559 generic.go:356] "Generic (PLEG): container finished" podID="b26a26f7-8e26-419d-abb1-2e793d5696d3" containerID="83a87e4d76daf76a1b77e0df38a98524fd9c2fd6222066680940c077da5e0375" exitCode=0 Jul 27 22:31:46.539302 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:46.538917 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" event={"ID":"b26a26f7-8e26-419d-abb1-2e793d5696d3","Type":"ContainerDied","Data":"83a87e4d76daf76a1b77e0df38a98524fd9c2fd6222066680940c077da5e0375"} Jul 27 22:31:47.672821 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:47.672791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:47.828276 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:47.828192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca\") pod \"b26a26f7-8e26-419d-abb1-2e793d5696d3\" (UID: \"b26a26f7-8e26-419d-abb1-2e793d5696d3\") " Jul 27 22:31:47.828540 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:47.828516 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b26a26f7-8e26-419d-abb1-2e793d5696d3" (UID: "b26a26f7-8e26-419d-abb1-2e793d5696d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:31:47.929552 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:47.929518 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26a26f7-8e26-419d-abb1-2e793d5696d3-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:31:48.550379 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:48.550350 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" Jul 27 22:31:48.550553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:48.550395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk" event={"ID":"b26a26f7-8e26-419d-abb1-2e793d5696d3","Type":"ContainerDied","Data":"5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf"} Jul 27 22:31:48.550553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:48.550439 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa1b8148746fe760cce00b57d2f2867a7a3d022c70b7ccf02144d7d82efcddf" Jul 27 22:31:55.436459 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:55.436429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:55.507283 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:55.507247 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:55.566907 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:55.566851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:59.760978 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:59.760931 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" podUID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" containerName="main" containerID="cri-o://965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b" gracePeriod=45 Jul 27 22:31:59.960528 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:59.960492 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:31:59.975322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:59.975251 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk"] Jul 27 22:31:59.977600 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:31:59.977575 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5bbdd-7l5jk"] Jul 27 22:32:00.014729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.014667 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:32:00.034203 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034329 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034329 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034270 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8nvq\" (UniqueName: \"kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034583 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034649 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034673 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir\") pod \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\" (UID: \"2049156d-b9ea-4df4-a3c5-9ad0737af5c6\") " Jul 27 22:32:00.034954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.034913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:00.035451 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.035055 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:00.035451 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.035051 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home" (OuterVolumeSpecName: "home") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:00.037122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.037077 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq" (OuterVolumeSpecName: "kube-api-access-c8nvq") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "kube-api-access-c8nvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:32:00.037122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.037079 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm" (OuterVolumeSpecName: "dshm") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:00.037367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.037135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2049156d-b9ea-4df4-a3c5-9ad0737af5c6" (UID: "2049156d-b9ea-4df4-a3c5-9ad0737af5c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:32:00.083819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.083787 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b26a26f7-8e26-419d-abb1-2e793d5696d3" path="/var/lib/kubelet/pods/b26a26f7-8e26-419d-abb1-2e793d5696d3/volumes" Jul 27 22:32:00.135700 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135664 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-dshm\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.135700 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135701 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tls-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.135908 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135716 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8nvq\" (UniqueName: \"kubernetes.io/projected/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-kube-api-access-c8nvq\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.135908 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135730 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-home\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.135908 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135742 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-model-cache\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.135908 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.135754 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2049156d-b9ea-4df4-a3c5-9ad0737af5c6-tmp-dir\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:00.348347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.348248 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:00.615679 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.615647 2559 generic.go:356] "Generic (PLEG): container finished" podID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" containerID="965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b" exitCode=0 Jul 27 22:32:00.615844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.615692 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" Jul 27 22:32:00.615844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.615699 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" event={"ID":"2049156d-b9ea-4df4-a3c5-9ad0737af5c6","Type":"ContainerDied","Data":"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b"} Jul 27 22:32:00.615844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.615732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp" event={"ID":"2049156d-b9ea-4df4-a3c5-9ad0737af5c6","Type":"ContainerDied","Data":"e911e5e7ba591fe15831afe0578fdf8f9ecc5eade9e6f3a36e6721f59583b875"} Jul 27 22:32:00.615844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.615747 2559 scope.go:117] "RemoveContainer" containerID="965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b" Jul 27 22:32:00.624320 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.624299 2559 scope.go:117] "RemoveContainer" containerID="965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b" Jul 27 22:32:00.624572 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:32:00.624552 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b\": container with ID starting with 965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b not found: ID does not exist" containerID="965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b" Jul 27 22:32:00.624641 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.624581 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b"} err="failed to get container status \"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b\": rpc error: code = NotFound desc = could not find container \"965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b\": container with ID starting with 965e03049132e0d29afe9da2b51864faee472e54e50b0156a5fd3c990784961b not found: ID does not exist" Jul 27 22:32:00.633686 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.633664 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:32:00.637535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:00.637514 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b2aa15/test-model-b2aa15-kserve-bc9bd6dbf-5gqqp"] Jul 27 22:32:02.083254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:02.083220 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" path="/var/lib/kubelet/pods/2049156d-b9ea-4df4-a3c5-9ad0737af5c6/volumes" Jul 27 22:32:06.584663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:06.584619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:06.649221 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:06.649183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:06.817806 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:06.817774 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:32:06.818010 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:06.817989 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" containerID="cri-o://53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc" gracePeriod=30 Jul 27 22:32:06.842314 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:06.842223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:07.052972 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:07.052921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:07.575229 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:07.575190 2559 patch_prober.go:28] interesting pod/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:32:07.575442 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:07.575246 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:32:12.068090 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.068061 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:32:12.140042 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140010 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140067 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140103 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140245 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140245 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140263 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140356 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140360 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:12.140455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140406 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t25s2\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2\") pod \"41583211-0aac-4a7d-bdd9-bc446331e6cf\" (UID: \"41583211-0aac-4a7d-bdd9-bc446331e6cf\") " Jul 27 22:32:12.140705 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140538 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:12.140705 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140610 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:12.140705 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:32:12.140868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140782 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41583211-0aac-4a7d-bdd9-bc446331e6cf-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.140868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140798 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.140868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140811 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.140868 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140824 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.141075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.140968 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:12.142778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.142745 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:32:12.143049 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.143027 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:32:12.143183 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.143137 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2" (OuterVolumeSpecName: "kube-api-access-t25s2") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "kube-api-access-t25s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:32:12.143183 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.143141 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41583211-0aac-4a7d-bdd9-bc446331e6cf" (UID: "41583211-0aac-4a7d-bdd9-bc446331e6cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:12.241527 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.241488 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t25s2\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-kube-api-access-t25s2\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.241527 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.241519 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.241527 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.241528 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.241527 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.241536 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.241794 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.241543 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41583211-0aac-4a7d-bdd9-bc446331e6cf-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:12.683602 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.683572 2559 generic.go:356] "Generic (PLEG): container finished" podID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerID="53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc" exitCode=0 Jul 27 22:32:12.683782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.683627 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" Jul 27 22:32:12.683782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.683660 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerDied","Data":"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc"} Jul 27 22:32:12.683782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.683714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j" event={"ID":"41583211-0aac-4a7d-bdd9-bc446331e6cf","Type":"ContainerDied","Data":"5201d0a3758d33e17736f715497e3b9d32470e1a893ba8eb0653a313e5dfc39f"} Jul 27 22:32:12.683782 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.683737 2559 scope.go:117] "RemoveContainer" containerID="53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc" Jul 27 22:32:12.692936 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.692921 2559 scope.go:117] "RemoveContainer" containerID="3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf" Jul 27 22:32:12.701212 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.701196 2559 scope.go:117] "RemoveContainer" containerID="53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc" Jul 27 22:32:12.701465 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:32:12.701447 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc\": container with ID starting with 53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc not found: ID does not exist" containerID="53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc" Jul 27 22:32:12.701508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.701477 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc"} err="failed to get container status \"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc\": rpc error: code = NotFound desc = could not find container \"53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc\": container with ID starting with 53fe6932db64ae3bdfd3fd490c6c6714607c33944dcaaa689a4c6d943d948ffc not found: ID does not exist" Jul 27 22:32:12.701508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.701495 2559 scope.go:117] "RemoveContainer" containerID="3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf" Jul 27 22:32:12.701763 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:32:12.701741 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf\": container with ID starting with 3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf not found: ID does not exist" containerID="3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf" Jul 27 22:32:12.701829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.701774 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf"} err="failed to get container status \"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf\": rpc error: code = NotFound desc = could not find container \"3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf\": container with ID starting with 3c418a8ea9f7e4bef71463557ec015de5aa378a83f666a11af971570e629bdcf not found: ID does not exist" Jul 27 22:32:12.706090 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.706065 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:32:12.709738 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:12.709720 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5bbdd-openshift-default-7b4d475c88-4kf7j"] Jul 27 22:32:14.083916 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:14.083883 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" path="/var/lib/kubelet/pods/41583211-0aac-4a7d-bdd9-bc446331e6cf/volumes" Jul 27 22:32:17.639257 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:17.639215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:17.709369 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:17.709334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:17.776668 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:17.776621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:22.250233 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:22.250196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:22.610054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:22.610022 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:28.679418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:28.679384 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:28.747013 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:28.746986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:28.906084 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:28.906047 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:32:28.906347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:28.906322 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" containerID="cri-o://76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8" gracePeriod=30 Jul 27 22:32:28.932607 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:28.932518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:29.168367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.168315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:29.483418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.483386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:29.512273 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.512235 2559 patch_prober.go:28] interesting pod/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:32:29.512415 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.512304 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:32:29.753175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753076 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753411 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753424 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753431 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b26a26f7-8e26-419d-abb1-2e793d5696d3" containerName="revoke-keys" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753436 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b26a26f7-8e26-419d-abb1-2e793d5696d3" containerName="revoke-keys" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753452 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" containerName="main" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753457 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" containerName="main" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753466 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753471 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753517 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753523 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2049156d-b9ea-4df4-a3c5-9ad0737af5c6" containerName="main" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753530 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="41583211-0aac-4a7d-bdd9-bc446331e6cf" containerName="istio-proxy" Jul 27 22:32:29.753558 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.753534 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b26a26f7-8e26-419d-abb1-2e793d5696d3" containerName="revoke-keys" Jul 27 22:32:29.756834 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.756815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.760643 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.760616 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-35e234-openshift-default-dockercfg-dr6m4\"" Jul 27 22:32:29.769128 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.769102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:32:29.790249 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.790220 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:29.892065 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892065 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j76l\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.892309 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.892307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993540 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993540 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993742 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993742 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993742 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993742 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993742 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993940 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.993940 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.993773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j76l\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.994473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.994406 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.994473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.994434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.994473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.994407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.994797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.994579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.994797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.994643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.996123 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.996096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:29.996270 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:29.996252 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:30.001749 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.001731 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j76l\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:30.002024 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.002004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token\") pod \"e2e-api-a-35e234-openshift-default-9f969b844-ldjch\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:30.066935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.066847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:30.067296 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.067265 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:30.211031 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.211003 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:32:30.213142 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:32:30.213106 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f37596_d933_4257_a73c_6daff0bde78a.slice/crio-f6ee2b66643e57096b6fd4620bf8ffcaed21535c1f3f7b00450541e908a21097 WatchSource:0}: Error finding container f6ee2b66643e57096b6fd4620bf8ffcaed21535c1f3f7b00450541e908a21097: Status 404 returned error can't find the container with id f6ee2b66643e57096b6fd4620bf8ffcaed21535c1f3f7b00450541e908a21097 Jul 27 22:32:30.215299 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.215259 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:30.215404 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.215353 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:30.215404 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.215397 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:30.774598 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.774561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerStarted","Data":"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827"} Jul 27 22:32:30.775095 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.774625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerStarted","Data":"f6ee2b66643e57096b6fd4620bf8ffcaed21535c1f3f7b00450541e908a21097"} Jul 27 22:32:30.797785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:30.797731 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podStartSLOduration=1.797715222 podStartE2EDuration="1.797715222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:32:30.796609654 +0000 UTC m=+2271.177681577" watchObservedRunningTime="2026-07-27 22:32:30.797715222 +0000 UTC m=+2271.178787145" Jul 27 22:32:31.067486 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:31.067389 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:31.072373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:31.072346 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:31.779535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:31.779507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:31.780471 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:31.780451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:32.732353 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:32.732312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:32.829332 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:32.829274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:32.900006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:32.899971 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:33.236815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:33.236783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:33.426011 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:33.425975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:33.522806 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:33.522717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:34.166917 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.166856 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:32:34.333808 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333771 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333819 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333845 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333866 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333906 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b2zw\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333960 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.333984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.333984 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.334371 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334059 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs\") pod \"ce438ec2-7176-4677-a6a7-3103522bb4a4\" (UID: \"ce438ec2-7176-4677-a6a7-3103522bb4a4\") " Jul 27 22:32:34.334371 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334126 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:34.334371 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:34.334524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334381 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.334524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334402 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.334524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334432 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:32:34.334663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.334599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:34.335040 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.335013 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:34.336293 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.336273 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:32:34.336375 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.336325 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:32:34.336827 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.336803 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:32:34.336827 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.336805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw" (OuterVolumeSpecName: "kube-api-access-4b2zw") pod "ce438ec2-7176-4677-a6a7-3103522bb4a4" (UID: "ce438ec2-7176-4677-a6a7-3103522bb4a4"). InnerVolumeSpecName "kube-api-access-4b2zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435195 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435223 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435231 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435239 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce438ec2-7176-4677-a6a7-3103522bb4a4-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435250 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b2zw\" (UniqueName: \"kubernetes.io/projected/ce438ec2-7176-4677-a6a7-3103522bb4a4-kube-api-access-4b2zw\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435259 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.435281 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.435267 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce438ec2-7176-4677-a6a7-3103522bb4a4-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:34.795445 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.795356 2559 generic.go:356] "Generic (PLEG): container finished" podID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerID="76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8" exitCode=0 Jul 27 22:32:34.795445 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.795415 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" Jul 27 22:32:34.795445 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.795439 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerDied","Data":"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8"} Jul 27 22:32:34.795667 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.795512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb" event={"ID":"ce438ec2-7176-4677-a6a7-3103522bb4a4","Type":"ContainerDied","Data":"292b36be9420973be5a83e5570d4279bf17402fd0b59c0c5906a318e4190e751"} Jul 27 22:32:34.795667 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.795533 2559 scope.go:117] "RemoveContainer" containerID="76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8" Jul 27 22:32:34.805191 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.805166 2559 scope.go:117] "RemoveContainer" containerID="fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac" Jul 27 22:32:34.813294 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.813268 2559 scope.go:117] "RemoveContainer" containerID="76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8" Jul 27 22:32:34.813538 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:32:34.813520 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8\": container with ID starting with 76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8 not found: ID does not exist" containerID="76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8" Jul 27 22:32:34.813595 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.813546 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8"} err="failed to get container status \"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8\": rpc error: code = NotFound desc = could not find container \"76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8\": container with ID starting with 76c64e7a888bfda65b2948d89d7dfb238869336a2ad08a2094550fb5a00037c8 not found: ID does not exist" Jul 27 22:32:34.813595 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.813561 2559 scope.go:117] "RemoveContainer" containerID="fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac" Jul 27 22:32:34.813786 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:32:34.813772 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac\": container with ID starting with fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac not found: ID does not exist" containerID="fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac" Jul 27 22:32:34.813824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.813791 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac"} err="failed to get container status \"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac\": rpc error: code = NotFound desc = could not find container \"fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac\": container with ID starting with fc9af5af697f6475da55e5c4b75a6a2deca1e0c8ed961cb1eb2568e6ea87bdac not found: ID does not exist" Jul 27 22:32:34.818114 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.818088 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:32:34.819897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:34.819878 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2aa15-openshift-default-6596bd897b-xkhmb"] Jul 27 22:32:36.084254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:36.084226 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" path="/var/lib/kubelet/pods/ce438ec2-7176-4677-a6a7-3103522bb4a4/volumes" Jul 27 22:32:37.219890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.219855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:37.684848 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.684811 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:37.942779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.942751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:32:37.943111 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.943094 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:37.943164 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.943116 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:37.943275 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.943259 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:37.947642 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.947625 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:37.950537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.950520 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-04ee43-openshift-default-dockercfg-64crh\"" Jul 27 22:32:37.959252 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.959229 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:32:37.979480 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:37.979447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:38.069813 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzrs\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069937 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.069992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.069974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.070233 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.070037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.070233 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.070072 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzrs\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171655 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.171772 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171747 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.172128 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.171934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.172128 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.172050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.172374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.172346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.172587 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.172566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.172664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.172570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.174272 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.174257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.174402 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.174386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.180664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.180631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzrs\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.180920 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.180900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token\") pod \"e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.259382 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.259298 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:38.289656 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.289590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:38.421789 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.421765 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:32:38.424297 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:32:38.424262 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b687484_93b8_4ae2_a3bb_68ca14b07074.slice/crio-adbe72e27ea94e71837108c49056de2fd07f6620b2141d0cff4709620bf75d7f WatchSource:0}: Error finding container adbe72e27ea94e71837108c49056de2fd07f6620b2141d0cff4709620bf75d7f: Status 404 returned error can't find the container with id adbe72e27ea94e71837108c49056de2fd07f6620b2141d0cff4709620bf75d7f Jul 27 22:32:38.426450 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.426423 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:38.426548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.426485 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:38.426548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.426516 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:38.819964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.819925 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerStarted","Data":"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e"} Jul 27 22:32:38.820127 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.819992 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerStarted","Data":"adbe72e27ea94e71837108c49056de2fd07f6620b2141d0cff4709620bf75d7f"} Jul 27 22:32:38.839907 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:38.839853 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podStartSLOduration=1.839834878 podStartE2EDuration="1.839834878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:32:38.838338402 +0000 UTC m=+2279.219410327" watchObservedRunningTime="2026-07-27 22:32:38.839834878 +0000 UTC m=+2279.220906802" Jul 27 22:32:39.259706 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:39.259668 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:39.264523 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:39.264504 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:39.826319 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:39.826279 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:39.827385 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:39.827356 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:40.810668 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:40.810628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:40.914421 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:40.914386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:40.997325 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:40.997288 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:41.096287 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:41.096183 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:41.528631 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:41.528587 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:41.627683 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:41.627645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:45.194995 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:45.194959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:47.821371 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:47.821329 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 22:32:47.821764 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:47.821432 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 22:32:48.264576 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.264497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:48.366001 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.365964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:32:48.874140 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.874107 2559 generic.go:356] "Generic (PLEG): container finished" podID="70f37596-d933-4257-a73c-6daff0bde78a" containerID="17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827" exitCode=0 Jul 27 22:32:48.874543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.874189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerDied","Data":"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827"} Jul 27 22:32:48.874586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.874578 2559 scope.go:117] "RemoveContainer" containerID="17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827" Jul 27 22:32:48.875602 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.875571 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:48.875714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.875665 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:48.875778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:48.875715 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:49.885753 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:49.885703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerStarted","Data":"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da"} Jul 27 22:32:50.067658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:50.067623 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:50.067837 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:50.067729 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 27 22:32:50.067837 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:50.067764 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 27 22:32:51.067548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:51.067511 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 27 22:32:51.067946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:51.067592 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 27 22:32:52.067452 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:52.067408 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 27 22:32:52.067670 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:52.067466 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 27 22:32:53.067399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.067355 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 27 22:32:53.067585 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.067432 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 27 22:32:53.164489 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.164457 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf"] Jul 27 22:32:53.164840 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.164784 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:53.164840 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.164795 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:53.164916 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.164851 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce438ec2-7176-4677-a6a7-3103522bb4a4" containerName="istio-proxy" Jul 27 22:32:53.167332 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.167315 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:53.169524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.169502 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 22:32:53.169646 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.169578 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9rxw\"" Jul 27 22:32:53.174655 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.174631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf"] Jul 27 22:32:53.224178 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.222642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35e234-vvnjf\" (UID: \"8c3ad609-14ec-4adc-9146-648149f79626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:53.323714 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.323601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35e234-vvnjf\" (UID: \"8c3ad609-14ec-4adc-9146-648149f79626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:53.324836 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.324811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35e234-vvnjf\" (UID: \"8c3ad609-14ec-4adc-9146-648149f79626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:53.480105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.480055 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:53.698193 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.693076 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf"] Jul 27 22:32:53.717963 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:32:53.717923 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ad609_14ec_4adc_9146_648149f79626.slice/crio-016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7 WatchSource:0}: Error finding container 016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7: Status 404 returned error can't find the container with id 016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7 Jul 27 22:32:53.917321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.917288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" event={"ID":"8c3ad609-14ec-4adc-9146-648149f79626","Type":"ContainerStarted","Data":"86b742cd6e39935e1a624c994d681a896597553619006178364a7c23f32c4f2c"} Jul 27 22:32:53.917485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.917343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" event={"ID":"8c3ad609-14ec-4adc-9146-648149f79626","Type":"ContainerStarted","Data":"016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7"} Jul 27 22:32:53.933623 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:53.933562 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" podStartSLOduration=0.933540677 podStartE2EDuration="933.540677ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:32:53.931311644 +0000 UTC m=+2294.312383569" watchObservedRunningTime="2026-07-27 22:32:53.933540677 +0000 UTC m=+2294.314612602" Jul 27 22:32:54.071707 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:54.071677 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:54.072065 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:54.072046 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:54.072842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:54.072826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:32:54.925507 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:54.925466 2559 generic.go:356] "Generic (PLEG): container finished" podID="8c3ad609-14ec-4adc-9146-648149f79626" containerID="86b742cd6e39935e1a624c994d681a896597553619006178364a7c23f32c4f2c" exitCode=0 Jul 27 22:32:54.925980 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:54.925561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" event={"ID":"8c3ad609-14ec-4adc-9146-648149f79626","Type":"ContainerDied","Data":"86b742cd6e39935e1a624c994d681a896597553619006178364a7c23f32c4f2c"} Jul 27 22:32:55.868387 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:55.868345 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 22:32:55.868581 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:55.868423 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 22:32:56.067352 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.067327 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:56.150811 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.150722 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca\") pod \"8c3ad609-14ec-4adc-9146-648149f79626\" (UID: \"8c3ad609-14ec-4adc-9146-648149f79626\") " Jul 27 22:32:56.151084 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.151063 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c3ad609-14ec-4adc-9146-648149f79626" (UID: "8c3ad609-14ec-4adc-9146-648149f79626"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:32:56.252274 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.252229 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ad609-14ec-4adc-9146-648149f79626-maas-api-service-ca\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:32:56.936652 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.936620 2559 generic.go:356] "Generic (PLEG): container finished" podID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerID="cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e" exitCode=0 Jul 27 22:32:56.936890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.936696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerDied","Data":"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e"} Jul 27 22:32:56.937135 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.937117 2559 scope.go:117] "RemoveContainer" containerID="cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e" Jul 27 22:32:56.938022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" Jul 27 22:32:56.938099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf" event={"ID":"8c3ad609-14ec-4adc-9146-648149f79626","Type":"ContainerDied","Data":"016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7"} Jul 27 22:32:56.938180 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938123 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016bf58933fe07c747b99325af90f03081de03550e489ef02095d4e43670c8c7" Jul 27 22:32:56.938252 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938227 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:56.938312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938297 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:56.938367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:56.938331 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:32:57.944445 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:57.944404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerStarted","Data":"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72"} Jul 27 22:32:58.259751 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:58.259655 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:32:58.259941 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:58.259921 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:32:58.260013 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:58.259964 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:32:59.260325 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:59.260285 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:32:59.260701 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:32:59.260348 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:33:00.259741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:00.259699 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:33:00.259958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:00.259772 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:33:01.260697 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:01.260660 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 27 22:33:01.261085 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:01.260721 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 27 22:33:02.263231 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:02.263199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:33:02.263605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:02.263542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:33:02.264253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:02.264232 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:33:08.885859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:08.885822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:08.958444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:08.958404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:09.024607 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:09.024577 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:13.284953 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:13.284921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:13.293159 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:13.293113 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf"] Jul 27 22:33:13.298671 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:13.298647 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35e234-vvnjf"] Jul 27 22:33:13.668635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:13.668601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:14.084434 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:14.084359 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3ad609-14ec-4adc-9146-648149f79626" path="/var/lib/kubelet/pods/8c3ad609-14ec-4adc-9146-648149f79626/volumes" Jul 27 22:33:19.882067 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:19.882035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:19.983119 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:19.983077 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:20.134489 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:20.134401 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:33:20.134645 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:20.134608 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" containerID="cri-o://2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da" gracePeriod=30 Jul 27 22:33:20.161663 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:20.161628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:20.386090 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:20.385994 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:24.074031 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:24.073993 2559 patch_prober.go:28] interesting pod/e2e-api-a-35e234-openshift-default-9f969b844-ldjch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:33:24.074460 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:24.074058 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:33:25.377724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.377699 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:33:25.515236 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515236 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515196 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515236 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515220 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515246 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515282 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515305 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515349 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515436 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j76l\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515463 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy\") pod \"70f37596-d933-4257-a73c-6daff0bde78a\" (UID: \"70f37596-d933-4257-a73c-6daff0bde78a\") " Jul 27 22:33:25.515535 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:25.515946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515608 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:25.515946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:33:25.515946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515732 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.515946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515751 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.515946 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.515766 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.516406 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.516094 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data" (OuterVolumeSpecName: "istio-data") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:25.517658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.517631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:25.517777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.517730 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:33:25.518164 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.518122 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l" (OuterVolumeSpecName: "kube-api-access-2j76l") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "kube-api-access-2j76l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:33:25.518164 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.518134 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token" (OuterVolumeSpecName: "istio-token") pod "70f37596-d933-4257-a73c-6daff0bde78a" (UID: "70f37596-d933-4257-a73c-6daff0bde78a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:33:25.616733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616696 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j76l\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-kube-api-access-2j76l\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.616733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616728 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.616733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616737 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f37596-d933-4257-a73c-6daff0bde78a-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.616733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616745 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f37596-d933-4257-a73c-6daff0bde78a-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.617011 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616754 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f37596-d933-4257-a73c-6daff0bde78a-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:25.617011 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:25.616761 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f37596-d933-4257-a73c-6daff0bde78a-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:26.085622 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.085574 2559 generic.go:356] "Generic (PLEG): container finished" podID="70f37596-d933-4257-a73c-6daff0bde78a" containerID="2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da" exitCode=0 Jul 27 22:33:26.085622 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.085610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerDied","Data":"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da"} Jul 27 22:33:26.085892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.085642 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" Jul 27 22:33:26.085892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.085680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch" event={"ID":"70f37596-d933-4257-a73c-6daff0bde78a","Type":"ContainerDied","Data":"f6ee2b66643e57096b6fd4620bf8ffcaed21535c1f3f7b00450541e908a21097"} Jul 27 22:33:26.085892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.085711 2559 scope.go:117] "RemoveContainer" containerID="2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da" Jul 27 22:33:26.096130 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.096107 2559 scope.go:117] "RemoveContainer" containerID="17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827" Jul 27 22:33:26.104650 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.104632 2559 scope.go:117] "RemoveContainer" containerID="2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da" Jul 27 22:33:26.104925 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:33:26.104906 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da\": container with ID starting with 2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da not found: ID does not exist" containerID="2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da" Jul 27 22:33:26.104980 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.104936 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da"} err="failed to get container status \"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da\": rpc error: code = NotFound desc = could not find container \"2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da\": container with ID starting with 2a41ae0111808d8ab8c6883cca0197eb56c856b6093fb9661316a4ff1d6669da not found: ID does not exist" Jul 27 22:33:26.104980 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.104956 2559 scope.go:117] "RemoveContainer" containerID="17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827" Jul 27 22:33:26.105224 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:33:26.105210 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827\": container with ID starting with 17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827 not found: ID does not exist" containerID="17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827" Jul 27 22:33:26.105262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.105227 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827"} err="failed to get container status \"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827\": rpc error: code = NotFound desc = could not find container \"17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827\": container with ID starting with 17dfccf26059fbbc2b1cf9f0b2c99a61e259782c191c4721f9461f43aa6a3827 not found: ID does not exist" Jul 27 22:33:26.110768 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.110732 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:33:26.114815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:26.114793 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-35e234-openshift-default-9f969b844-ldjch"] Jul 27 22:33:28.083514 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:28.083480 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f37596-d933-4257-a73c-6daff0bde78a" path="/var/lib/kubelet/pods/70f37596-d933-4257-a73c-6daff0bde78a/volumes" Jul 27 22:33:36.095215 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:36.095099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:37.449834 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:37.449784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:40.275383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:40.275352 2559 scope.go:117] "RemoveContainer" containerID="0f9e7c219229f4e76575e265d131c0498f5a67673fd879bfcff970dc3a562d81" Jul 27 22:33:40.568367 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:40.568271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:40.964819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:40.964787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:46.953959 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:46.953920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:47.016852 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.016828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:47.230689 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.230604 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:33:47.230848 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.230809 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" containerID="cri-o://a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72" gracePeriod=30 Jul 27 22:33:47.262819 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.262787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:47.265275 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.265244 2559 patch_prober.go:28] interesting pod/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:33:47.265393 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.265336 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:33:47.494683 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.494593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:47.825485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:47.825395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:48.129637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129606 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129888 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129897 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129906 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129912 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129923 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3ad609-14ec-4adc-9146-648149f79626" containerName="revoke-keys" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.129929 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3ad609-14ec-4adc-9146-648149f79626" containerName="revoke-keys" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.130003 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.130009 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3ad609-14ec-4adc-9146-648149f79626" containerName="revoke-keys" Jul 27 22:33:48.130022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.130015 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f37596-d933-4257-a73c-6daff0bde78a" containerName="istio-proxy" Jul 27 22:33:48.131790 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.131774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.134431 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.134409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c5cbe3-openshift-default-dockercfg-jz4p8\"" Jul 27 22:33:48.151220 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.151185 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:33:48.174668 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.174635 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:48.197078 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197040 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197178 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtl7\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197225 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.197533 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.197291 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtl7\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298722 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298721 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.298879 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.298863 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.300762 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.300735 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.300859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.300826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.310776 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.310752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.312005 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.311988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtl7\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7\") pod \"e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.442000 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.441910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:48.496811 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.496765 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:48.586619 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.586582 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:33:48.589877 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:33:48.589842 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a595e5_8e3d_47d2_bc9f_fb4e03852078.slice/crio-f7cc134219b3bd91686a9e98246f26408cef7b62d6cfc7b733e305f31e8fc5a2 WatchSource:0}: Error finding container f7cc134219b3bd91686a9e98246f26408cef7b62d6cfc7b733e305f31e8fc5a2: Status 404 returned error can't find the container with id f7cc134219b3bd91686a9e98246f26408cef7b62d6cfc7b733e305f31e8fc5a2 Jul 27 22:33:48.591917 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.591887 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:48.591981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.591960 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:48.592013 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:48.591991 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:49.203506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:49.203472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerStarted","Data":"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47"} Jul 27 22:33:49.203966 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:49.203528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerStarted","Data":"f7cc134219b3bd91686a9e98246f26408cef7b62d6cfc7b733e305f31e8fc5a2"} Jul 27 22:33:49.224711 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:49.224639 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podStartSLOduration=1.22461643 podStartE2EDuration="1.22461643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:33:49.224052072 +0000 UTC m=+2349.605123997" watchObservedRunningTime="2026-07-27 22:33:49.22461643 +0000 UTC m=+2349.605688357" Jul 27 22:33:49.442687 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:49.442653 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:49.447957 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:49.447931 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:50.208430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:50.208404 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:50.209532 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:50.209506 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:33:51.117855 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:51.117821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:51.204076 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:51.204031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:51.278462 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:51.278426 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:51.377724 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:51.377624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:51.900322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:51.900291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:52.008708 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.008673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:52.476650 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.476629 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:33:52.535368 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535279 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535368 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535368 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzzrs\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535486 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535537 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535569 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535592 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.535658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token\") pod \"7b687484-93b8-4ae2-a3bb-68ca14b07074\" (UID: \"7b687484-93b8-4ae2-a3bb-68ca14b07074\") " Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535850 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535851 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535866 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535976 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.535994 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.536008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.536007 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.537036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.537006 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.537785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.537762 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:33:52.537884 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.537836 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:33:52.537884 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.537870 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs" (OuterVolumeSpecName: "kube-api-access-rzzrs") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "kube-api-access-rzzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:33:52.537992 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.537979 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b687484-93b8-4ae2-a3bb-68ca14b07074" (UID: "7b687484-93b8-4ae2-a3bb-68ca14b07074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:33:52.636844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636814 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.636844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636839 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.636844 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636848 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.637061 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636856 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b687484-93b8-4ae2-a3bb-68ca14b07074-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.637061 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636865 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzzrs\" (UniqueName: \"kubernetes.io/projected/7b687484-93b8-4ae2-a3bb-68ca14b07074-kube-api-access-rzzrs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:52.637061 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:52.636875 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b687484-93b8-4ae2-a3bb-68ca14b07074-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:33:53.226066 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.226033 2559 generic.go:356] "Generic (PLEG): container finished" podID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerID="a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72" exitCode=0 Jul 27 22:33:53.226254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.226082 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" Jul 27 22:33:53.226254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.226124 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerDied","Data":"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72"} Jul 27 22:33:53.226254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.226197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8" event={"ID":"7b687484-93b8-4ae2-a3bb-68ca14b07074","Type":"ContainerDied","Data":"adbe72e27ea94e71837108c49056de2fd07f6620b2141d0cff4709620bf75d7f"} Jul 27 22:33:53.226254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.226218 2559 scope.go:117] "RemoveContainer" containerID="a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72" Jul 27 22:33:53.241594 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.241574 2559 scope.go:117] "RemoveContainer" containerID="cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e" Jul 27 22:33:53.250073 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.250053 2559 scope.go:117] "RemoveContainer" containerID="a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72" Jul 27 22:33:53.250395 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:33:53.250374 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72\": container with ID starting with a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72 not found: ID does not exist" containerID="a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72" Jul 27 22:33:53.250459 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.250401 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72"} err="failed to get container status \"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72\": rpc error: code = NotFound desc = could not find container \"a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72\": container with ID starting with a025ad40f6dd9ad20196b5ea1647c436662719856d5617989da18ef5ba04be72 not found: ID does not exist" Jul 27 22:33:53.250459 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.250421 2559 scope.go:117] "RemoveContainer" containerID="cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e" Jul 27 22:33:53.250680 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:33:53.250658 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e\": container with ID starting with cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e not found: ID does not exist" containerID="cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e" Jul 27 22:33:53.250738 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.250690 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e"} err="failed to get container status \"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e\": rpc error: code = NotFound desc = could not find container \"cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e\": container with ID starting with cb07baf5638c34f123424b129b2b49c0632f7603bf0ee31bdce5693ab3d2de8e not found: ID does not exist" Jul 27 22:33:53.251734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.251716 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:33:53.257180 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:53.257143 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-04ee43-openshift-default-f54f68c5b-wpkn8"] Jul 27 22:33:54.083815 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:54.083766 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" path="/var/lib/kubelet/pods/7b687484-93b8-4ae2-a3bb-68ca14b07074/volumes" Jul 27 22:33:55.611557 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:55.611521 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:55.932890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:55.932792 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:56.202429 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.202341 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:33:56.202790 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.202769 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:33:56.202921 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.202794 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:33:56.202921 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.202884 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:33:56.205800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.205781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.208098 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.208078 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f36b41-openshift-default-dockercfg-m7qhf\"" Jul 27 22:33:56.217830 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.217809 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:33:56.229769 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.229726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:56.268909 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.268875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.268913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.268958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269022 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.269345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.269285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzdc\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzdc\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.370732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.371054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.370957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.371125 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.371105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.371208 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.371166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.371267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.371207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.371267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.371234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.373003 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.372986 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.373399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.373377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.382176 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.382129 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.382270 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.382222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzdc\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc\") pod \"e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.413718 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.413689 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:56.515908 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.515815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:56.659312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.659287 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:33:56.661386 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:33:56.661352 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2c91be_59e4_463d_9165_68d6d8c10fcd.slice/crio-ee2c0bbea972f5873c900c05f2efa82528679ff8d3096f78a6657841fbae99a6 WatchSource:0}: Error finding container ee2c0bbea972f5873c900c05f2efa82528679ff8d3096f78a6657841fbae99a6: Status 404 returned error can't find the container with id ee2c0bbea972f5873c900c05f2efa82528679ff8d3096f78a6657841fbae99a6 Jul 27 22:33:56.663382 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.663348 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:56.663484 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.663424 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:56.663553 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:56.663491 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:33:57.251043 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:57.251009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerStarted","Data":"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d"} Jul 27 22:33:57.251254 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:57.251065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerStarted","Data":"ee2c0bbea972f5873c900c05f2efa82528679ff8d3096f78a6657841fbae99a6"} Jul 27 22:33:57.275405 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:57.275359 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podStartSLOduration=1.275343383 podStartE2EDuration="1.275343383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:33:57.274052444 +0000 UTC m=+2357.655124369" watchObservedRunningTime="2026-07-27 22:33:57.275343383 +0000 UTC m=+2357.656415308" Jul 27 22:33:57.516999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:57.516913 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:57.521700 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:57.521671 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:58.257796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:58.257762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:58.258933 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:58.258916 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:33:59.184958 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.184923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:59.266468 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.266435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:59.337720 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.337674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:59.769537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.769499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:59.869729 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.869702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:33:59.969961 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:33:59.969928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:03.691034 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:03.691003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:08.319099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.318986 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerID="d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47" exitCode=0 Jul 27 22:34:08.319099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.319066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerDied","Data":"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47"} Jul 27 22:34:08.319577 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.319472 2559 scope.go:117] "RemoveContainer" containerID="d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47" Jul 27 22:34:08.320532 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.320503 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:08.320606 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.320591 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:08.320651 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:08.320639 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:09.329706 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:09.329663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerStarted","Data":"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d"} Jul 27 22:34:09.442444 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:09.442409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:34:09.442618 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:09.442507 2559 patch_prober.go:28] interesting pod/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:34:09.442618 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:09.442542 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:34:10.443361 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:10.443313 2559 patch_prober.go:28] interesting pod/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:34:10.443797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:10.443384 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:34:11.442554 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:11.442511 2559 patch_prober.go:28] interesting pod/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:34:11.442732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:11.442564 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:34:12.443247 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:12.443209 2559 patch_prober.go:28] interesting pod/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 27 22:34:12.443666 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:12.443278 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 27 22:34:13.445935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:13.445898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:34:13.446427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:13.446193 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:34:13.447071 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:13.447049 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:34:14.299328 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:14.299295 2559 patch_prober.go:28] interesting pod/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 22:34:14.299509 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:14.299360 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 22:34:15.361487 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.361457 2559 generic.go:356] "Generic (PLEG): container finished" podID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerID="38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d" exitCode=0 Jul 27 22:34:15.361982 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.361541 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerDied","Data":"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d"} Jul 27 22:34:15.361982 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.361921 2559 scope.go:117] "RemoveContainer" containerID="38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d" Jul 27 22:34:15.362873 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.362846 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:15.362931 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.362915 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:15.362966 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:15.362946 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:34:16.368785 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:16.368747 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerStarted","Data":"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d"} Jul 27 22:34:16.516625 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:16.516588 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:34:16.516829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:16.516803 2559 patch_prober.go:28] interesting pod/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:34:16.516911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:16.516867 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:34:17.516733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:17.516698 2559 patch_prober.go:28] interesting pod/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:34:17.517131 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:17.516768 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:34:18.517083 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:18.517049 2559 patch_prober.go:28] interesting pod/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:34:18.517595 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:18.517105 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:34:19.517047 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:19.517001 2559 patch_prober.go:28] interesting pod/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 27 22:34:19.517296 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:19.517070 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 27 22:34:20.520723 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:20.520692 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:34:20.521094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:20.520957 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:34:20.521627 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:20.521612 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:34:23.620007 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:23.619970 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:23.685572 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:23.685538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:25.257036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:25.256995 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:26.372787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:26.372729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:40.159483 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:40.159452 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:34:40.165454 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:40.165430 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:34:40.298975 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:40.298940 2559 scope.go:117] "RemoveContainer" containerID="c691c796c65045812ed8969adfa175908e9dd571e20dd15ab26e4498d4a16043" Jul 27 22:34:52.340859 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.340826 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:34:52.341260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.341122 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:34:52.341260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.341132 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:34:52.341260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.341227 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b687484-93b8-4ae2-a3bb-68ca14b07074" containerName="istio-proxy" Jul 27 22:34:52.344041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.344025 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.347207 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.347184 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"kube-root-ca.crt\"" Jul 27 22:34:52.347345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.347185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"auth-test-model-f36b41-kserve-self-signed-certs\"" Jul 27 22:34:52.347345 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.347184 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"default-dockercfg-mcrwb\"" Jul 27 22:34:52.347468 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.347215 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"openshift-service-ca.crt\"" Jul 27 22:34:52.353984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.353962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:34:52.365378 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.365347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:34:52.466106 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.466322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466136 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.466322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.466322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbzx\" (UniqueName: \"kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.466322 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.466512 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.466357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.566911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.566882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.566911 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.566915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.566951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbzx\" (UniqueName: \"kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.567436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.567408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.569493 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.569459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.569738 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.569722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.575420 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.575398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbzx\" (UniqueName: \"kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx\") pod \"auth-test-model-f36b41-kserve-54bb88487f-f582w\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.654132 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.654104 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:34:52.784915 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:52.784892 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:34:52.789850 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:34:52.789810 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd17378_d064_4f91_8c4a_100c409ba406.slice/crio-726453b7640ae3ed27673b8ade8f20d54b0a4960d3f6a9a7b9e302fdafb66239 WatchSource:0}: Error finding container 726453b7640ae3ed27673b8ade8f20d54b0a4960d3f6a9a7b9e302fdafb66239: Status 404 returned error can't find the container with id 726453b7640ae3ed27673b8ade8f20d54b0a4960d3f6a9a7b9e302fdafb66239 Jul 27 22:34:53.566470 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:53.566433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" event={"ID":"ebd17378-d064-4f91-8c4a-100c409ba406","Type":"ContainerStarted","Data":"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9"} Jul 27 22:34:53.566470 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:53.566472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" event={"ID":"ebd17378-d064-4f91-8c4a-100c409ba406","Type":"ContainerStarted","Data":"726453b7640ae3ed27673b8ade8f20d54b0a4960d3f6a9a7b9e302fdafb66239"} Jul 27 22:34:53.585260 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:34:53.585202 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" podStartSLOduration=1.585183302 podStartE2EDuration="1.585183302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:53.583994327 +0000 UTC m=+2413.965066252" watchObservedRunningTime="2026-07-27 22:34:53.585183302 +0000 UTC m=+2413.966255224" Jul 27 22:35:02.654522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:35:02.654446 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:35:02.654522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:35:02.654494 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:35:02.667395 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:35:02.667363 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:35:03.632108 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:35:03.632080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:35:45.726374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:35:45.726336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:38:40.346195 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:38:40.346138 2559 scope.go:117] "RemoveContainer" containerID="83a87e4d76daf76a1b77e0df38a98524fd9c2fd6222066680940c077da5e0375" Jul 27 22:39:20.445520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:20.445489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:39:21.368408 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:21.368373 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:39:21.619063 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:21.618986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:39:31.683545 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:31.683511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:39:36.387953 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.387883 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" podUID="ebd17378-d064-4f91-8c4a-100c409ba406" containerName="main" containerID="cri-o://a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9" gracePeriod=45 Jul 27 22:39:36.645491 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.645420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:39:36.691390 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691434 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691456 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjbzx\" (UniqueName: \"kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691495 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691514 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691563 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691531 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm\") pod \"ebd17378-d064-4f91-8c4a-100c409ba406\" (UID: \"ebd17378-d064-4f91-8c4a-100c409ba406\") " Jul 27 22:39:36.691839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:39:36.691839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691763 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home" (OuterVolumeSpecName: "home") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:39:36.691839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691772 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-tmp-dir\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.691839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.691789 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:39:36.693893 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.693868 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm" (OuterVolumeSpecName: "dshm") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:39:36.693893 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.693874 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx" (OuterVolumeSpecName: "kube-api-access-cjbzx") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "kube-api-access-cjbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:39:36.694019 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.693873 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebd17378-d064-4f91-8c4a-100c409ba406" (UID: "ebd17378-d064-4f91-8c4a-100c409ba406"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:39:36.792734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.792692 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-model-cache\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.792734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.792725 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-home\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.792734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.792734 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd17378-d064-4f91-8c4a-100c409ba406-dshm\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.792734 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.792742 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd17378-d064-4f91-8c4a-100c409ba406-tls-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.793006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.792752 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbzx\" (UniqueName: \"kubernetes.io/projected/ebd17378-d064-4f91-8c4a-100c409ba406-kube-api-access-cjbzx\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:39:36.987626 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.987532 2559 generic.go:356] "Generic (PLEG): container finished" podID="ebd17378-d064-4f91-8c4a-100c409ba406" containerID="a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9" exitCode=0 Jul 27 22:39:36.987626 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.987576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" Jul 27 22:39:36.987824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.987622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" event={"ID":"ebd17378-d064-4f91-8c4a-100c409ba406","Type":"ContainerDied","Data":"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9"} Jul 27 22:39:36.987824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.987676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w" event={"ID":"ebd17378-d064-4f91-8c4a-100c409ba406","Type":"ContainerDied","Data":"726453b7640ae3ed27673b8ade8f20d54b0a4960d3f6a9a7b9e302fdafb66239"} Jul 27 22:39:36.987824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.987698 2559 scope.go:117] "RemoveContainer" containerID="a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9" Jul 27 22:39:36.997023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.997009 2559 scope.go:117] "RemoveContainer" containerID="a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9" Jul 27 22:39:36.997296 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:39:36.997273 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9\": container with ID starting with a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9 not found: ID does not exist" containerID="a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9" Jul 27 22:39:36.997401 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:36.997303 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9"} err="failed to get container status \"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9\": rpc error: code = NotFound desc = could not find container \"a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9\": container with ID starting with a7b892b8ead6dbc420bb6e3fa7ecb6671837379da8af2599c6e69a3a5bf6ffe9 not found: ID does not exist" Jul 27 22:39:37.011301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:37.011279 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:39:37.014954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:37.014937 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/auth-test-model-f36b41-kserve-54bb88487f-f582w"] Jul 27 22:39:38.083630 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:38.083599 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd17378-d064-4f91-8c4a-100c409ba406" path="/var/lib/kubelet/pods/ebd17378-d064-4f91-8c4a-100c409ba406/volumes" Jul 27 22:39:40.191653 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:40.191617 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:39:40.198138 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:40.198116 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:39:40.366267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:39:40.366221 2559 scope.go:117] "RemoveContainer" containerID="86b742cd6e39935e1a624c994d681a896597553619006178364a7c23f32c4f2c" Jul 27 22:40:01.948436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.948400 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:40:01.949039 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.948706 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd17378-d064-4f91-8c4a-100c409ba406" containerName="main" Jul 27 22:40:01.949039 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.948716 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd17378-d064-4f91-8c4a-100c409ba406" containerName="main" Jul 27 22:40:01.949039 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.948774 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd17378-d064-4f91-8c4a-100c409ba406" containerName="main" Jul 27 22:40:01.951654 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.951626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:01.955099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.955078 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"openshift-service-ca.crt\"" Jul 27 22:40:01.955267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.955083 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"sub-test-model-f36b41-kserve-self-signed-certs\"" Jul 27 22:40:01.955267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.955129 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"default-dockercfg-mcrwb\"" Jul 27 22:40:01.955267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.955083 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"kube-root-ca.crt\"" Jul 27 22:40:01.961551 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:01.961529 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:40:02.076366 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.076335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:40:02.116025 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.115990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.116025 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.116026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.116265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.116042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.116265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.116066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.116265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.116209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.116366 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.116311 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xgf\" (UniqueName: \"kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.217800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xgf\" (UniqueName: \"kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.217800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.217800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.217800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.217800 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.218226 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.217923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.218301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.218224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.218301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.218288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.218411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.218345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.220205 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.220184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.220502 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.220487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.225768 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.225747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xgf\" (UniqueName: \"kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf\") pod \"sub-test-model-f36b41-kserve-69b6999dbc-76wh9\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.260824 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.260786 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:02.404952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:02.404924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:40:02.407296 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:40:02.407269 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae8f08e4_4db7_4c96_8f5c_42dfaddbcebd.slice/crio-27cbd3ab95719cb4438f471f975a88b86663cb0ed912cda9ac450834aec70dec WatchSource:0}: Error finding container 27cbd3ab95719cb4438f471f975a88b86663cb0ed912cda9ac450834aec70dec: Status 404 returned error can't find the container with id 27cbd3ab95719cb4438f471f975a88b86663cb0ed912cda9ac450834aec70dec Jul 27 22:40:03.119229 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:03.119192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" event={"ID":"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd","Type":"ContainerStarted","Data":"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75"} Jul 27 22:40:03.119229 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:03.119237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" event={"ID":"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd","Type":"ContainerStarted","Data":"27cbd3ab95719cb4438f471f975a88b86663cb0ed912cda9ac450834aec70dec"} Jul 27 22:40:03.141890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:03.141837 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" podStartSLOduration=2.141817053 podStartE2EDuration="2.141817053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:40:03.141047007 +0000 UTC m=+2723.522118925" watchObservedRunningTime="2026-07-27 22:40:03.141817053 +0000 UTC m=+2723.522888977" Jul 27 22:40:06.759243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:06.759210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:40:12.261288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:12.261247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:12.261783 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:12.261313 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:12.273811 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:12.273785 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:40:13.178059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:40:13.178031 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:41:09.943761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:09.943673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:41:10.852278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:10.852248 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:41:10.877041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:10.877006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:41:15.856890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:15.856850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:41:25.871480 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:25.871435 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" podUID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" containerName="main" containerID="cri-o://0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75" gracePeriod=45 Jul 27 22:41:26.120633 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.120608 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:41:26.238181 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238086 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238181 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238125 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238181 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238175 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238454 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xgf\" (UniqueName: \"kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238454 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238253 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238454 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238361 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home\") pod \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\" (UID: \"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd\") " Jul 27 22:41:26.238603 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238500 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:41:26.238683 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:41:26.238739 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238670 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tmp-dir\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.238784 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.238740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home" (OuterVolumeSpecName: "home") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:41:26.240363 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.240339 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm" (OuterVolumeSpecName: "dshm") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:41:26.241053 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.241029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:41:26.241122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.241051 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf" (OuterVolumeSpecName: "kube-api-access-v4xgf") pod "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" (UID: "ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd"). InnerVolumeSpecName "kube-api-access-v4xgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:41:26.339812 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.339780 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-model-cache\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.339812 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.339811 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-home\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.339812 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.339819 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-dshm\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.340105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.339828 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-tls-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.340105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.339837 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xgf\" (UniqueName: \"kubernetes.io/projected/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd-kube-api-access-v4xgf\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:41:26.523736 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.523633 2559 generic.go:356] "Generic (PLEG): container finished" podID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" containerID="0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75" exitCode=0 Jul 27 22:41:26.523736 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.523683 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" Jul 27 22:41:26.523736 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.523716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" event={"ID":"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd","Type":"ContainerDied","Data":"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75"} Jul 27 22:41:26.524000 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.523758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9" event={"ID":"ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd","Type":"ContainerDied","Data":"27cbd3ab95719cb4438f471f975a88b86663cb0ed912cda9ac450834aec70dec"} Jul 27 22:41:26.524000 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.523779 2559 scope.go:117] "RemoveContainer" containerID="0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75" Jul 27 22:41:26.533430 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.533412 2559 scope.go:117] "RemoveContainer" containerID="0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75" Jul 27 22:41:26.533659 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:41:26.533639 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75\": container with ID starting with 0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75 not found: ID does not exist" containerID="0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75" Jul 27 22:41:26.533709 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.533669 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75"} err="failed to get container status \"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75\": rpc error: code = NotFound desc = could not find container \"0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75\": container with ID starting with 0b79d2be39482aa2e41327e5ebf4d760ed7d595ca44308323ff15feddaf75c75 not found: ID does not exist" Jul 27 22:41:26.546112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.546090 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:41:26.552797 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:26.552778 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/sub-test-model-f36b41-kserve-69b6999dbc-76wh9"] Jul 27 22:41:28.083991 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:28.083960 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" path="/var/lib/kubelet/pods/ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd/volumes" Jul 27 22:41:33.939077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.939043 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:41:33.939479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.939412 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" containerName="main" Jul 27 22:41:33.939479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.939423 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" containerName="main" Jul 27 22:41:33.939549 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.939492 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8f08e4-4db7-4c96-8f5c-42dfaddbcebd" containerName="main" Jul 27 22:41:33.943919 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.943900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:33.946388 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.946371 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"kube-root-ca.crt\"" Jul 27 22:41:33.947207 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.947189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"openshift-service-ca.crt\"" Jul 27 22:41:33.947303 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.947236 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"default-dockercfg-mcrwb\"" Jul 27 22:41:33.947303 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.947253 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f36b41\"/\"rate-test-model-f36b41-kserve-self-signed-certs\"" Jul 27 22:41:33.956019 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.955996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:41:33.981246 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:33.981222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:41:34.007249 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.007373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.007373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.007373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65k6l\" (UniqueName: \"kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.007508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.007508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.007459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108363 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108751 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65k6l\" (UniqueName: \"kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108751 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108863 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108901 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.108954 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.108939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.110767 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.110745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.111107 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.111089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.116493 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.116468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65k6l\" (UniqueName: \"kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l\") pod \"rate-test-model-f36b41-kserve-796ddcff59-wrtc4\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.253637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.253550 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:34.383560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.383538 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:41:34.385550 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:41:34.385514 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd246644c_d0ab_40b7_a438_3ecf1471184c.slice/crio-c032072c51b74efef937b94d7de57fc6307097cf696084aa64c6f6172e00c722 WatchSource:0}: Error finding container c032072c51b74efef937b94d7de57fc6307097cf696084aa64c6f6172e00c722: Status 404 returned error can't find the container with id c032072c51b74efef937b94d7de57fc6307097cf696084aa64c6f6172e00c722 Jul 27 22:41:34.563384 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.563296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" event={"ID":"d246644c-d0ab-40b7-a438-3ecf1471184c","Type":"ContainerStarted","Data":"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913"} Jul 27 22:41:34.563384 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.563335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" event={"ID":"d246644c-d0ab-40b7-a438-3ecf1471184c","Type":"ContainerStarted","Data":"c032072c51b74efef937b94d7de57fc6307097cf696084aa64c6f6172e00c722"} Jul 27 22:41:34.581122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:34.581065 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" podStartSLOduration=1.581046014 podStartE2EDuration="1.581046014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:41:34.580700717 +0000 UTC m=+2814.961772641" watchObservedRunningTime="2026-07-27 22:41:34.581046014 +0000 UTC m=+2814.962117940" Jul 27 22:41:44.254511 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:44.254471 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:44.255103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:44.254525 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:44.266576 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:44.266552 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:41:44.622542 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:41:44.622515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:42:53.617708 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:53.617668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:42:54.588702 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:54.588664 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:42:54.610804 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:54.610767 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:42:55.493142 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.493108 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-140677006776496"] Jul 27 22:42:55.496346 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.496330 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:55.498732 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.498712 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gh969\"" Jul 27 22:42:55.503333 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.503027 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006776496"] Jul 27 22:42:55.609628 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.609594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhjv\" (UniqueName: \"kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv\") pod \"test-curl-7588-140677006776496\" (UID: \"84ee2ab2-f327-48ba-a857-63d692df218a\") " pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:55.710580 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.710547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhjv\" (UniqueName: \"kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv\") pod \"test-curl-7588-140677006776496\" (UID: \"84ee2ab2-f327-48ba-a857-63d692df218a\") " pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:55.718516 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.718489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhjv\" (UniqueName: \"kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv\") pod \"test-curl-7588-140677006776496\" (UID: \"84ee2ab2-f327-48ba-a857-63d692df218a\") " pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:55.806394 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.806309 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:55.936656 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.936633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006776496"] Jul 27 22:42:55.938635 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:42:55.938604 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84ee2ab2_f327_48ba_a857_63d692df218a.slice/crio-aff0364adf89228967822bd7ff41a9c69c36f66804a053686322a330e4947cf5 WatchSource:0}: Error finding container aff0364adf89228967822bd7ff41a9c69c36f66804a053686322a330e4947cf5: Status 404 returned error can't find the container with id aff0364adf89228967822bd7ff41a9c69c36f66804a053686322a330e4947cf5 Jul 27 22:42:55.940095 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.940074 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:42:55.960691 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:55.960663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006776496" event={"ID":"84ee2ab2-f327-48ba-a857-63d692df218a","Type":"ContainerStarted","Data":"aff0364adf89228967822bd7ff41a9c69c36f66804a053686322a330e4947cf5"} Jul 27 22:42:56.966897 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:56.966861 2559 generic.go:356] "Generic (PLEG): container finished" podID="84ee2ab2-f327-48ba-a857-63d692df218a" containerID="aca14b7d720ffc5f69c19e4324b9f84cb17f1a20943ebb40b215a238cc3bd272" exitCode=0 Jul 27 22:42:56.967314 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:56.966928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006776496" event={"ID":"84ee2ab2-f327-48ba-a857-63d692df218a","Type":"ContainerDied","Data":"aca14b7d720ffc5f69c19e4324b9f84cb17f1a20943ebb40b215a238cc3bd272"} Jul 27 22:42:58.106427 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.106404 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:58.133761 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.133732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhjv\" (UniqueName: \"kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv\") pod \"84ee2ab2-f327-48ba-a857-63d692df218a\" (UID: \"84ee2ab2-f327-48ba-a857-63d692df218a\") " Jul 27 22:42:58.135104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.135077 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7588-140677006776496_84ee2ab2-f327-48ba-a857-63d692df218a/test-curl-7588-140677006776496/0.log" Jul 27 22:42:58.136112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.136090 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv" (OuterVolumeSpecName: "kube-api-access-vhhjv") pod "84ee2ab2-f327-48ba-a857-63d692df218a" (UID: "84ee2ab2-f327-48ba-a857-63d692df218a"). InnerVolumeSpecName "kube-api-access-vhhjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:42:58.161477 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.161446 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-140677006776496"] Jul 27 22:42:58.163508 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.163488 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-140677006776496"] Jul 27 22:42:58.234698 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.234602 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhjv\" (UniqueName: \"kubernetes.io/projected/84ee2ab2-f327-48ba-a857-63d692df218a-kube-api-access-vhhjv\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:42:58.978569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.978539 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff0364adf89228967822bd7ff41a9c69c36f66804a053686322a330e4947cf5" Jul 27 22:42:58.978569 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:42:58.978542 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:42:58.987951 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:42:58.987922 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7588-140677006776496\" is forbidden: User \"system:node:ip-10-0-134-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-139.ec2.internal' and this object" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:43:00.084355 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:43:00.084312 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7588-140677006776496\" is forbidden: User \"system:node:ip-10-0-134-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-139.ec2.internal' and this object" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" pod="opendatahub/test-curl-7588-140677006776496" Jul 27 22:43:00.085082 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:00.085061 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" path="/var/lib/kubelet/pods/84ee2ab2-f327-48ba-a857-63d692df218a/volumes" Jul 27 22:43:04.181258 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.181217 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-140677006985088"] Jul 27 22:43:04.181740 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.181661 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" containerName="test-curl-7588-140677006776496" Jul 27 22:43:04.181740 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.181679 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" containerName="test-curl-7588-140677006776496" Jul 27 22:43:04.181852 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.181759 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ee2ab2-f327-48ba-a857-63d692df218a" containerName="test-curl-7588-140677006776496" Jul 27 22:43:04.184913 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.184895 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:04.187330 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.187313 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gh969\"" Jul 27 22:43:04.192902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.192883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006985088"] Jul 27 22:43:04.286288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.286255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c6n\" (UniqueName: \"kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n\") pod \"test-curl-7588-140677006985088\" (UID: \"747cc133-656f-4e1f-965f-f52d96917daa\") " pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:04.386831 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.386794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c6n\" (UniqueName: \"kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n\") pod \"test-curl-7588-140677006985088\" (UID: \"747cc133-656f-4e1f-965f-f52d96917daa\") " pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:04.397796 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.397760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c6n\" (UniqueName: \"kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n\") pod \"test-curl-7588-140677006985088\" (UID: \"747cc133-656f-4e1f-965f-f52d96917daa\") " pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:04.494823 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.494737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:04.627927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:04.627894 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006985088"] Jul 27 22:43:04.629949 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:43:04.629913 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod747cc133_656f_4e1f_965f_f52d96917daa.slice/crio-9c92d9ee0010c8082f3b6012911dcd35fb0df02e202212e96b56c7baa49b2062 WatchSource:0}: Error finding container 9c92d9ee0010c8082f3b6012911dcd35fb0df02e202212e96b56c7baa49b2062: Status 404 returned error can't find the container with id 9c92d9ee0010c8082f3b6012911dcd35fb0df02e202212e96b56c7baa49b2062 Jul 27 22:43:05.011073 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:05.011043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006985088" event={"ID":"747cc133-656f-4e1f-965f-f52d96917daa","Type":"ContainerStarted","Data":"9c92d9ee0010c8082f3b6012911dcd35fb0df02e202212e96b56c7baa49b2062"} Jul 27 22:43:05.029036 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:05.028990 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7588-140677006985088" podStartSLOduration=0.733399162 podStartE2EDuration="1.028974387s" totalImagesPullingTime="295.575225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:43:05.027712286 +0000 UTC m=+2905.408784213" watchObservedRunningTime="2026-07-27 22:43:05.028974387 +0000 UTC m=+2905.410046359" Jul 27 22:43:05.128549 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:05.128495 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7588-140677006985088_747cc133-656f-4e1f-965f-f52d96917daa/test-curl-7588-140677006985088/0.log" Jul 27 22:43:06.017200 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:06.017143 2559 generic.go:356] "Generic (PLEG): container finished" podID="747cc133-656f-4e1f-965f-f52d96917daa" containerID="75f1150ed2a76e5f066fbf4eb55b6726d748c3c0b6bd9ceb12613cf81cba860b" exitCode=0 Jul 27 22:43:06.017200 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:06.017183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006985088" event={"ID":"747cc133-656f-4e1f-965f-f52d96917daa","Type":"ContainerDied","Data":"75f1150ed2a76e5f066fbf4eb55b6726d748c3c0b6bd9ceb12613cf81cba860b"} Jul 27 22:43:07.146560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.146536 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:07.188561 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.184744 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-140677006985088"] Jul 27 22:43:07.191816 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.191784 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-140677006985088"] Jul 27 22:43:07.212939 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.212904 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24c6n\" (UniqueName: \"kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n\") pod \"747cc133-656f-4e1f-965f-f52d96917daa\" (UID: \"747cc133-656f-4e1f-965f-f52d96917daa\") " Jul 27 22:43:07.215102 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.215074 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n" (OuterVolumeSpecName: "kube-api-access-24c6n") pod "747cc133-656f-4e1f-965f-f52d96917daa" (UID: "747cc133-656f-4e1f-965f-f52d96917daa"). InnerVolumeSpecName "kube-api-access-24c6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:43:07.309422 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.309339 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:07.309703 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.309688 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747cc133-656f-4e1f-965f-f52d96917daa" containerName="test-curl-7588-140677006985088" Jul 27 22:43:07.309750 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.309705 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="747cc133-656f-4e1f-965f-f52d96917daa" containerName="test-curl-7588-140677006985088" Jul 27 22:43:07.309783 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.309770 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="747cc133-656f-4e1f-965f-f52d96917daa" containerName="test-curl-7588-140677006985088" Jul 27 22:43:07.312510 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.312494 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:07.313777 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.313755 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24c6n\" (UniqueName: \"kubernetes.io/projected/747cc133-656f-4e1f-965f-f52d96917daa-kube-api-access-24c6n\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:07.319134 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.319107 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:07.414399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.414355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc\") pod \"test-curl-7588-140677006820960\" (UID: \"82f73222-ca41-4b1f-b36f-f6e2263a7ba6\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:07.515873 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.515829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc\") pod \"test-curl-7588-140677006820960\" (UID: \"82f73222-ca41-4b1f-b36f-f6e2263a7ba6\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:07.527044 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.527023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc\") pod \"test-curl-7588-140677006820960\" (UID: \"82f73222-ca41-4b1f-b36f-f6e2263a7ba6\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:07.622348 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.622317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:07.752947 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:07.752919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:07.754799 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:43:07.754777 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82f73222_ca41_4b1f_b36f_f6e2263a7ba6.slice/crio-b14e712592413bc14f2cca6b4b9946b3928c916237207e41a12275ed40796b45 WatchSource:0}: Error finding container b14e712592413bc14f2cca6b4b9946b3928c916237207e41a12275ed40796b45: Status 404 returned error can't find the container with id b14e712592413bc14f2cca6b4b9946b3928c916237207e41a12275ed40796b45 Jul 27 22:43:08.027984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.027910 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:08.027984 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.027931 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c92d9ee0010c8082f3b6012911dcd35fb0df02e202212e96b56c7baa49b2062" Jul 27 22:43:08.029136 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.029114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006820960" event={"ID":"82f73222-ca41-4b1f-b36f-f6e2263a7ba6","Type":"ContainerStarted","Data":"b14e712592413bc14f2cca6b4b9946b3928c916237207e41a12275ed40796b45"} Jul 27 22:43:08.038646 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:43:08.038619 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7588-140677006985088\" is forbidden: User \"system:node:ip-10-0-134-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-139.ec2.internal' and this object" podUID="747cc133-656f-4e1f-965f-f52d96917daa" pod="opendatahub/test-curl-7588-140677006985088" Jul 27 22:43:08.083041 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.083015 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747cc133-656f-4e1f-965f-f52d96917daa" path="/var/lib/kubelet/pods/747cc133-656f-4e1f-965f-f52d96917daa/volumes" Jul 27 22:43:08.576820 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.576788 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:08.676740 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:08.676709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:09.035228 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.035190 2559 generic.go:356] "Generic (PLEG): container finished" podID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" containerID="64c7534fb08ab343f9b1c7c7695834ec5ff76efdf8f8991dbf0dd4a7be1ff11a" exitCode=0 Jul 27 22:43:09.035399 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.035271 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006820960" event={"ID":"82f73222-ca41-4b1f-b36f-f6e2263a7ba6","Type":"ContainerDied","Data":"64c7534fb08ab343f9b1c7c7695834ec5ff76efdf8f8991dbf0dd4a7be1ff11a"} Jul 27 22:43:09.607471 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.607427 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" podUID="d246644c-d0ab-40b7-a438-3ecf1471184c" containerName="main" containerID="cri-o://88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913" gracePeriod=45 Jul 27 22:43:09.859171 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.859113 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:43:09.936976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.936941 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65k6l\" (UniqueName: \"kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.936976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.936980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.937271 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.937271 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937028 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.937271 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937051 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.937271 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937108 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm\") pod \"d246644c-d0ab-40b7-a438-3ecf1471184c\" (UID: \"d246644c-d0ab-40b7-a438-3ecf1471184c\") " Jul 27 22:43:09.937477 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937313 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:43:09.937477 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937445 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-model-cache\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:09.937567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937473 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home" (OuterVolumeSpecName: "home") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:43:09.937567 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.937485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:43:09.939670 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.939634 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l" (OuterVolumeSpecName: "kube-api-access-65k6l") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "kube-api-access-65k6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:43:09.939871 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.939846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm" (OuterVolumeSpecName: "dshm") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:43:09.940021 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:09.939999 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d246644c-d0ab-40b7-a438-3ecf1471184c" (UID: "d246644c-d0ab-40b7-a438-3ecf1471184c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:43:10.037888 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.037858 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-tmp-dir\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.037888 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.037883 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-home\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.037888 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.037893 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d246644c-d0ab-40b7-a438-3ecf1471184c-dshm\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.038120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.037900 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65k6l\" (UniqueName: \"kubernetes.io/projected/d246644c-d0ab-40b7-a438-3ecf1471184c-kube-api-access-65k6l\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.038120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.037908 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d246644c-d0ab-40b7-a438-3ecf1471184c-tls-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.041852 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.041818 2559 generic.go:356] "Generic (PLEG): container finished" podID="d246644c-d0ab-40b7-a438-3ecf1471184c" containerID="88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913" exitCode=0 Jul 27 22:43:10.042014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.041857 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" Jul 27 22:43:10.042014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.041908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" event={"ID":"d246644c-d0ab-40b7-a438-3ecf1471184c","Type":"ContainerDied","Data":"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913"} Jul 27 22:43:10.042014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.041950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4" event={"ID":"d246644c-d0ab-40b7-a438-3ecf1471184c","Type":"ContainerDied","Data":"c032072c51b74efef937b94d7de57fc6307097cf696084aa64c6f6172e00c722"} Jul 27 22:43:10.042014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.041968 2559 scope.go:117] "RemoveContainer" containerID="88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913" Jul 27 22:43:10.052728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.052706 2559 scope.go:117] "RemoveContainer" containerID="88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913" Jul 27 22:43:10.053015 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:43:10.052996 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913\": container with ID starting with 88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913 not found: ID does not exist" containerID="88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913" Jul 27 22:43:10.053087 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.053030 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913"} err="failed to get container status \"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913\": rpc error: code = NotFound desc = could not find container \"88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913\": container with ID starting with 88893bc6a42f88a42b7a8730ec2f813c20577c438cc10c1fd58a2987925a7913 not found: ID does not exist" Jul 27 22:43:10.066310 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.066269 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:43:10.070759 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.070725 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f36b41/rate-test-model-f36b41-kserve-796ddcff59-wrtc4"] Jul 27 22:43:10.085408 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.085381 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d246644c-d0ab-40b7-a438-3ecf1471184c" path="/var/lib/kubelet/pods/d246644c-d0ab-40b7-a438-3ecf1471184c/volumes" Jul 27 22:43:10.177326 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.177299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.200672 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.200643 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7588-140677006820960_82f73222-ca41-4b1f-b36f-f6e2263a7ba6/test-curl-7588-140677006820960/0.log" Jul 27 22:43:10.225123 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.225093 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:10.231004 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.230981 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:10.239162 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.239128 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc\") pod \"82f73222-ca41-4b1f-b36f-f6e2263a7ba6\" (UID: \"82f73222-ca41-4b1f-b36f-f6e2263a7ba6\") " Jul 27 22:43:10.241268 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.241247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc" (OuterVolumeSpecName: "kube-api-access-zcvpc") pod "82f73222-ca41-4b1f-b36f-f6e2263a7ba6" (UID: "82f73222-ca41-4b1f-b36f-f6e2263a7ba6"). InnerVolumeSpecName "kube-api-access-zcvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:43:10.340647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.340609 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvpc\" (UniqueName: \"kubernetes.io/projected/82f73222-ca41-4b1f-b36f-f6e2263a7ba6-kube-api-access-zcvpc\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:10.343473 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343428 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:10.343802 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343788 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d246644c-d0ab-40b7-a438-3ecf1471184c" containerName="main" Jul 27 22:43:10.343851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343804 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d246644c-d0ab-40b7-a438-3ecf1471184c" containerName="main" Jul 27 22:43:10.343851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343813 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" containerName="test-curl-7588-140677006820960" Jul 27 22:43:10.343851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343819 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" containerName="test-curl-7588-140677006820960" Jul 27 22:43:10.343964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343883 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d246644c-d0ab-40b7-a438-3ecf1471184c" containerName="main" Jul 27 22:43:10.343964 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.343893 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" containerName="test-curl-7588-140677006820960" Jul 27 22:43:10.346816 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.346801 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.351348 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.351326 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:10.442182 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.442050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qvl\" (UniqueName: \"kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl\") pod \"test-curl-7588-140677006820960\" (UID: \"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.542798 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.542759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qvl\" (UniqueName: \"kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl\") pod \"test-curl-7588-140677006820960\" (UID: \"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.552442 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.552418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qvl\" (UniqueName: \"kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl\") pod \"test-curl-7588-140677006820960\" (UID: \"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3\") " pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.656361 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.656321 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:10.787522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:10.787498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:10.789603 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:43:10.789569 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2270c4d_1a64_42f5_ad27_9cf2aa16a7d3.slice/crio-68d200dc0892401c149e8dac29022169e6f1b5106ebd35966b675962baef6ff9 WatchSource:0}: Error finding container 68d200dc0892401c149e8dac29022169e6f1b5106ebd35966b675962baef6ff9: Status 404 returned error can't find the container with id 68d200dc0892401c149e8dac29022169e6f1b5106ebd35966b675962baef6ff9 Jul 27 22:43:11.051396 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:11.051320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006820960" event={"ID":"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3","Type":"ContainerStarted","Data":"68d200dc0892401c149e8dac29022169e6f1b5106ebd35966b675962baef6ff9"} Jul 27 22:43:11.052455 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:11.052432 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14e712592413bc14f2cca6b4b9946b3928c916237207e41a12275ed40796b45" Jul 27 22:43:11.052548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:11.052464 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:11.065234 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:11.065200 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7588-140677006820960" oldPodUID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" Jul 27 22:43:12.059616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:12.059517 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" containerID="7d147215c3ea88ba7a5a5a60b6feef0c166fbf2485a78bdcfa55e57585976c89" exitCode=0 Jul 27 22:43:12.059616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:12.059609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-140677006820960" event={"ID":"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3","Type":"ContainerDied","Data":"7d147215c3ea88ba7a5a5a60b6feef0c166fbf2485a78bdcfa55e57585976c89"} Jul 27 22:43:12.083872 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:12.083843 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f73222-ca41-4b1f-b36f-f6e2263a7ba6" path="/var/lib/kubelet/pods/82f73222-ca41-4b1f-b36f-f6e2263a7ba6/volumes" Jul 27 22:43:13.202413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.202386 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:13.223450 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.223421 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7588-140677006820960_a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3/test-curl-7588-140677006820960/0.log" Jul 27 22:43:13.245104 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.245048 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:13.247237 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.247209 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-140677006820960"] Jul 27 22:43:13.267167 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.267118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4qvl\" (UniqueName: \"kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl\") pod \"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3\" (UID: \"a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3\") " Jul 27 22:43:13.269365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.269336 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl" (OuterVolumeSpecName: "kube-api-access-t4qvl") pod "a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" (UID: "a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3"). InnerVolumeSpecName "kube-api-access-t4qvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:43:13.368293 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.368263 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qvl\" (UniqueName: \"kubernetes.io/projected/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3-kube-api-access-t4qvl\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:43:13.751990 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:13.751901 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:14.029409 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.029331 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:43:14.029658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.029645 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" containerName="test-curl-7588-140677006820960" Jul 27 22:43:14.029658 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.029658 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" containerName="test-curl-7588-140677006820960" Jul 27 22:43:14.029781 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.029721 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" containerName="test-curl-7588-140677006820960" Jul 27 22:43:14.032664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.032646 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.034989 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.034972 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-48ceef-openshift-default-dockercfg-vrd6g\"" Jul 27 22:43:14.044220 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.044198 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:43:14.062620 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.062579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:14.071175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.071131 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d200dc0892401c149e8dac29022169e6f1b5106ebd35966b675962baef6ff9" Jul 27 22:43:14.071175 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.071177 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:14.082434 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:43:14.082406 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7588-140677006820960\" is forbidden: User \"system:node:ip-10-0-134-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-139.ec2.internal' and this object" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" pod="opendatahub/test-curl-7588-140677006820960" Jul 27 22:43:14.083879 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.083858 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3" path="/var/lib/kubelet/pods/a2270c4d-1a64-42f5-ad27-9cf2aa16a7d3/volumes" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqnf\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.179822 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.175623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.276851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.276795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.276851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.276861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.276884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.276899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.276925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277075 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqnf\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.277902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.277873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.279511 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.279458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.279737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.279686 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.289972 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.289951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqnf\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.290560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.290541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token\") pod \"e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.343580 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.343546 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:14.484359 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.484334 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:43:14.485939 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:43:14.485913 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbd7c06_6c92_4710_99a2_51d9f2a53ccd.slice/crio-5aecb8338ad4dda7e5bea5d92044a625c3d34d1516df05f0801d59b78a3013d2 WatchSource:0}: Error finding container 5aecb8338ad4dda7e5bea5d92044a625c3d34d1516df05f0801d59b78a3013d2: Status 404 returned error can't find the container with id 5aecb8338ad4dda7e5bea5d92044a625c3d34d1516df05f0801d59b78a3013d2 Jul 27 22:43:14.488109 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.488075 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:14.488215 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.488190 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:14.488288 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:14.488237 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:15.078287 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.078251 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerStarted","Data":"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279"} Jul 27 22:43:15.078495 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.078313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerStarted","Data":"5aecb8338ad4dda7e5bea5d92044a625c3d34d1516df05f0801d59b78a3013d2"} Jul 27 22:43:15.100125 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.100047 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podStartSLOduration=1.100026682 podStartE2EDuration="1.100026682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:43:15.098892923 +0000 UTC m=+2915.479964846" watchObservedRunningTime="2026-07-27 22:43:15.100026682 +0000 UTC m=+2915.481098609" Jul 27 22:43:15.253349 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.253316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:15.344781 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.344695 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:15.350009 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:15.349984 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:16.084312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:16.084280 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:16.085347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:16.085329 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:17.136022 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:17.135989 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:17.249488 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:17.249456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:17.346101 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:17.346066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:17.453203 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:17.453099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:18.010574 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:18.010542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:18.107590 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:18.107558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:21.602401 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:21.602365 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:32.510882 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.510844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:32.818325 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.818237 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:43:32.824355 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.824332 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.826927 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.826903 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e47da1-openshift-default-dockercfg-f5tmb\"" Jul 27 22:43:32.832466 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.832401 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:43:32.847779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.847779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7r4\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848017 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.847990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.848017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.848324 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.848067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.851297 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.851274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:32.948841 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.949052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.948979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.949915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.949003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7r4\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.950127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.950097 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.950227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.950679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.955172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.952469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.957881 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.957853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.958850 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.958823 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.959284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.959244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:32.960494 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:32.960471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7r4\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4\") pod \"e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:33.136038 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.136002 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:33.283540 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.283508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:43:33.284777 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:43:33.284750 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f742c66_ea01_422c_9134_676adc457ded.slice/crio-84b2f25c83cbdea8a9053987d613d71469620f965f99686e1f4b39c3ca123b4b WatchSource:0}: Error finding container 84b2f25c83cbdea8a9053987d613d71469620f965f99686e1f4b39c3ca123b4b: Status 404 returned error can't find the container with id 84b2f25c83cbdea8a9053987d613d71469620f965f99686e1f4b39c3ca123b4b Jul 27 22:43:33.287165 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.287113 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:33.287246 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.287228 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:33.287287 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.287276 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:33.701054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:33.701024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:34.180257 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.180215 2559 generic.go:356] "Generic (PLEG): container finished" podID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerID="14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279" exitCode=0 Jul 27 22:43:34.180488 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.180290 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerDied","Data":"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279"} Jul 27 22:43:34.180749 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.180730 2559 scope.go:117] "RemoveContainer" containerID="14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279" Jul 27 22:43:34.181930 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.181898 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:34.182052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.181971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerStarted","Data":"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648"} Jul 27 22:43:34.182052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.182012 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerStarted","Data":"84b2f25c83cbdea8a9053987d613d71469620f965f99686e1f4b39c3ca123b4b"} Jul 27 22:43:34.182052 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.181974 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:34.182243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.182072 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:34.223118 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:34.223053 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podStartSLOduration=2.223026821 podStartE2EDuration="2.223026821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:43:34.220562136 +0000 UTC m=+2934.601634061" watchObservedRunningTime="2026-07-27 22:43:34.223026821 +0000 UTC m=+2934.604098746" Jul 27 22:43:35.137242 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.137200 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:35.142219 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.142191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:35.194105 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.194065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerStarted","Data":"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22"} Jul 27 22:43:35.194571 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.194543 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:35.198977 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.198947 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:35.344253 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.344214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:35.344564 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.344533 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 27 22:43:35.344723 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:35.344600 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 27 22:43:36.151763 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.151734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:36.239408 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.239379 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:36.317265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.317231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:36.344051 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.344015 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 27 22:43:36.344363 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.344330 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 27 22:43:36.776099 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.776057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:36.880178 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.878823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:36.983000 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:36.982962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:37.344921 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:37.344878 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 27 22:43:37.345350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:37.344938 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 27 22:43:38.344912 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:38.344882 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 27 22:43:38.345089 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:38.344936 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 27 22:43:39.348027 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:39.347992 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:39.348479 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:39.348278 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:39.349055 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:39.349041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:43:40.615588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:40.615553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:51.812100 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:51.812071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:43:55.317682 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.317625 2559 generic.go:356] "Generic (PLEG): container finished" podID="5f742c66-ea01-422c-9134-676adc457ded" containerID="bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648" exitCode=0 Jul 27 22:43:55.318107 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.317701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerDied","Data":"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648"} Jul 27 22:43:55.318107 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.318085 2559 scope.go:117] "RemoveContainer" containerID="bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648" Jul 27 22:43:55.319321 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.319293 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:55.319397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.319376 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:55.319440 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:55.319427 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:43:56.325045 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:56.325015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerStarted","Data":"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16"} Jul 27 22:43:57.136912 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:57.136868 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:43:57.137120 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:57.137095 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 27 22:43:57.137224 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:57.137202 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 27 22:43:58.137097 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:58.137055 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 27 22:43:58.137543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:58.137146 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 27 22:43:59.136829 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:59.136792 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 27 22:43:59.137010 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:43:59.136858 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 27 22:44:00.136883 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:00.136849 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 27 22:44:00.137312 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:00.136915 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 27 22:44:01.141059 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:01.140975 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:44:01.141560 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:01.141385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:44:01.141981 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:01.141962 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:44:03.770371 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:03.770338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:44:40.226093 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:40.226065 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:44:40.233080 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:44:40.233058 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:45:00.428583 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.428527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:00.727548 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.727469 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:00.730835 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.730819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.733359 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.733335 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ea5c1d-openshift-default-dockercfg-srv22\"" Jul 27 22:45:00.743733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.743714 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:00.752770 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.752872 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrkh\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.752872 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.752872 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752863 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.753020 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.753020 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.753020 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.753020 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.752971 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.753243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.753056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853655 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrkh\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853777 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.853842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854282 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.853866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854282 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.854214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854372 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.854291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854372 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.854310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854488 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.854468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.854534 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.854482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.856335 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.856307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.856524 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.856501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.862528 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.862507 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrkh\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:00.862673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:00.862655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token\") pod \"e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:01.040351 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.040256 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:01.189500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.189460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:01.192130 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:45:01.192103 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebb3bb8_d2d9_402d_b69e_4cbb25f2aa36.slice/crio-c7e0179d29c27bed161f3070ca42a4631981f169834b54d76fe6f57d52bf8db4 WatchSource:0}: Error finding container c7e0179d29c27bed161f3070ca42a4631981f169834b54d76fe6f57d52bf8db4: Status 404 returned error can't find the container with id c7e0179d29c27bed161f3070ca42a4631981f169834b54d76fe6f57d52bf8db4 Jul 27 22:45:01.194746 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.194715 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:01.194813 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.194789 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:01.194851 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.194817 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:01.672097 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.672063 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerStarted","Data":"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8"} Jul 27 22:45:01.672588 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.672118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerStarted","Data":"c7e0179d29c27bed161f3070ca42a4631981f169834b54d76fe6f57d52bf8db4"} Jul 27 22:45:01.694130 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:01.694057 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podStartSLOduration=1.694035695 podStartE2EDuration="1.694035695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:45:01.693261249 +0000 UTC m=+3022.074333173" watchObservedRunningTime="2026-07-27 22:45:01.694035695 +0000 UTC m=+3022.075107623" Jul 27 22:45:02.040650 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:02.040554 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:02.045502 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:02.045477 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:02.677664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:02.677636 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:02.678493 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:02.678470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:03.424920 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:03.424893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:03.611839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:03.611804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:03.711802 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:03.711719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:03.870952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:03.870895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:03.968952 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:03.968862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:04.382171 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:04.382120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:04.494949 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:04.494919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:08.065029 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:08.064996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:12.470522 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:12.470488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:16.751696 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.751660 2559 generic.go:356] "Generic (PLEG): container finished" podID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerID="0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8" exitCode=0 Jul 27 22:45:16.752186 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.751736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerDied","Data":"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8"} Jul 27 22:45:16.752186 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.752115 2559 scope.go:117] "RemoveContainer" containerID="0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8" Jul 27 22:45:16.753237 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.753203 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:16.753295 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.753281 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:16.753331 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:16.753311 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:45:17.683372 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:17.683324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:17.759243 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:17.759201 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerStarted","Data":"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2"} Jul 27 22:45:18.041080 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:18.040987 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:18.042197 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:18.042172 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 27 22:45:18.042317 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:18.042232 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 27 22:45:19.041030 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:19.040995 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 27 22:45:19.041461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:19.041065 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 27 22:45:20.041298 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:20.041264 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 27 22:45:20.041674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:20.041319 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 27 22:45:21.040976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:21.040934 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 27 22:45:21.041197 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:21.041014 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 27 22:45:22.045439 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:22.045406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:22.045846 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:22.045682 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:22.046316 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:22.046295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:29.566336 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:29.566306 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:29.644231 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:29.644193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:29.713613 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:29.713581 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:34.024063 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:34.024031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:34.388779 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:34.388742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:40.331999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.331967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:40.432565 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.432536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:40.590179 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.590082 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:40.590339 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.590308 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" containerID="cri-o://fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2" gracePeriod=30 Jul 27 22:45:40.608000 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.607972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:40.823871 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:40.823835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:45.839902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.839879 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:45.908086 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.908051 2559 generic.go:356] "Generic (PLEG): container finished" podID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerID="fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2" exitCode=0 Jul 27 22:45:45.908284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.908102 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" Jul 27 22:45:45.908284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.908137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerDied","Data":"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2"} Jul 27 22:45:45.908284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.908209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm" event={"ID":"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36","Type":"ContainerDied","Data":"c7e0179d29c27bed161f3070ca42a4631981f169834b54d76fe6f57d52bf8db4"} Jul 27 22:45:45.908284 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.908233 2559 scope.go:117] "RemoveContainer" containerID="fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2" Jul 27 22:45:45.917184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.917167 2559 scope.go:117] "RemoveContainer" containerID="0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8" Jul 27 22:45:45.927190 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.927120 2559 scope.go:117] "RemoveContainer" containerID="fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2" Jul 27 22:45:45.927542 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:45:45.927518 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2\": container with ID starting with fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2 not found: ID does not exist" containerID="fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2" Jul 27 22:45:45.927604 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.927557 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2"} err="failed to get container status \"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2\": rpc error: code = NotFound desc = could not find container \"fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2\": container with ID starting with fd277c69ec0aff80b571df6e4824eb69afc2b492f46f13bc856f284ba09e89b2 not found: ID does not exist" Jul 27 22:45:45.927604 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.927580 2559 scope.go:117] "RemoveContainer" containerID="0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8" Jul 27 22:45:45.927830 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:45:45.927812 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8\": container with ID starting with 0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8 not found: ID does not exist" containerID="0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8" Jul 27 22:45:45.927887 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.927838 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8"} err="failed to get container status \"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8\": rpc error: code = NotFound desc = could not find container \"0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8\": container with ID starting with 0a63fd6a78e23dec7212afe1501f16d5e58f315ad7e706ec7fc79b831336a2e8 not found: ID does not exist" Jul 27 22:45:45.981266 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981275 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffrkh\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981325 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981374 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981446 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981418 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981450 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981477 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981518 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data\") pod \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\" (UID: \"4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36\") " Jul 27 22:45:45.981737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981539 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:45:45.981737 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981663 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:45:45.982184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981764 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:45:45.982184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:45:45.982184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981891 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:45.982184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981907 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:45.982184 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.981920 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:45.982469 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.982442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:45:45.983708 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.983677 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh" (OuterVolumeSpecName: "kube-api-access-ffrkh") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "kube-api-access-ffrkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:45:45.983810 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.983728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:45:45.984218 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.984200 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:45:45.984279 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:45.984210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" (UID: "4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:45:46.083133 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083101 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrkh\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-kube-api-access-ffrkh\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.083133 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083135 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.083350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083168 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.083350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083182 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.083350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083194 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.083350 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.083207 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:45:46.225460 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.225381 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:46.232006 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:46.231983 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ea5c1d-openshift-default-5968867498-2sgfm"] Jul 27 22:45:48.084472 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:48.084432 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" path="/var/lib/kubelet/pods/4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36/volumes" Jul 27 22:45:59.544501 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:59.544462 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:59.622845 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:59.622808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:45:59.685707 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:45:59.685676 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:04.121652 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:04.121610 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:04.496924 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:04.496837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:09.563635 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:09.563603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:45.942273 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:45.942235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:46.688464 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:46.688431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:50.138931 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:50.138896 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:52.159215 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.159180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:52.222134 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.222103 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:52.413506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.413415 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:46:52.413684 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.413661 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" containerID="cri-o://85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22" gracePeriod=30 Jul 27 22:46:52.435818 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.435782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:52.650835 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:52.650803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:46:54.351126 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:54.351090 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:46:54.351517 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:54.351176 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:46:57.661223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.661201 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:46:57.722463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722387 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722431 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722483 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722538 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722580 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722603 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722626 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722656 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722721 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kqnf\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf\") pod \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\" (UID: \"adbd7c06-6c92-4710-99a2-51d9f2a53ccd\") " Jul 27 22:46:57.722955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722722 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:46:57.722955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:46:57.722955 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.722941 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:46:57.723265 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723075 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:46:57.723323 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723265 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.723323 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723285 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.723323 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723299 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.723323 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723313 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.723533 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.723514 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data" (OuterVolumeSpecName: "istio-data") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:46:57.725510 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.725483 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token" (OuterVolumeSpecName: "istio-token") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:46:57.725510 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.725501 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf" (OuterVolumeSpecName: "kube-api-access-2kqnf") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "kube-api-access-2kqnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:46:57.725674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.725618 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:46:57.725674 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.725624 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "adbd7c06-6c92-4710-99a2-51d9f2a53ccd" (UID: "adbd7c06-6c92-4710-99a2-51d9f2a53ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:46:57.824137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.824106 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.824137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.824130 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.824137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.824139 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.824137 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.824146 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:57.824431 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:57.824174 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kqnf\" (UniqueName: \"kubernetes.io/projected/adbd7c06-6c92-4710-99a2-51d9f2a53ccd-kube-api-access-2kqnf\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:46:58.280935 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.280900 2559 generic.go:356] "Generic (PLEG): container finished" podID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerID="85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22" exitCode=0 Jul 27 22:46:58.281122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.280962 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" Jul 27 22:46:58.281122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.280981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerDied","Data":"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22"} Jul 27 22:46:58.281122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.281046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4" event={"ID":"adbd7c06-6c92-4710-99a2-51d9f2a53ccd","Type":"ContainerDied","Data":"5aecb8338ad4dda7e5bea5d92044a625c3d34d1516df05f0801d59b78a3013d2"} Jul 27 22:46:58.281122 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.281072 2559 scope.go:117] "RemoveContainer" containerID="85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22" Jul 27 22:46:58.291374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.291352 2559 scope.go:117] "RemoveContainer" containerID="14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279" Jul 27 22:46:58.299214 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.299188 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:46:58.301356 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.301340 2559 scope.go:117] "RemoveContainer" containerID="85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22" Jul 27 22:46:58.301703 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:46:58.301681 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22\": container with ID starting with 85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22 not found: ID does not exist" containerID="85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22" Jul 27 22:46:58.301778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.301716 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22"} err="failed to get container status \"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22\": rpc error: code = NotFound desc = could not find container \"85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22\": container with ID starting with 85ffa46c9992d7f3bf81cc8c2f2a7d70a1ea30eaeab932ae497d1701bb874b22 not found: ID does not exist" Jul 27 22:46:58.301778 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.301735 2559 scope.go:117] "RemoveContainer" containerID="14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279" Jul 27 22:46:58.302014 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:46:58.301999 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279\": container with ID starting with 14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279 not found: ID does not exist" containerID="14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279" Jul 27 22:46:58.302054 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.302019 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279"} err="failed to get container status \"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279\": rpc error: code = NotFound desc = could not find container \"14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279\": container with ID starting with 14415fc00b7bec4cafd383baa5a281e94167d89d34e23fb2188ac1c976119279 not found: ID does not exist" Jul 27 22:46:58.302728 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:46:58.302709 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-48ceef-openshift-default-67b9b48948-6bdn4"] Jul 27 22:47:00.084074 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:00.084036 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" path="/var/lib/kubelet/pods/adbd7c06-6c92-4710-99a2-51d9f2a53ccd/volumes" Jul 27 22:47:03.256621 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:03.256549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:03.323549 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:03.323519 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:03.387612 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:03.387575 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:07.848230 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:07.848194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:08.236082 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:08.236043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:14.016989 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.016953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:14.117389 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.117360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:14.275877 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.275793 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:47:14.276047 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.276021 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" containerID="cri-o://e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16" gracePeriod=30 Jul 27 22:47:14.297374 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.297349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:14.502684 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:14.502652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:16.143839 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:16.143801 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:47:16.144248 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:16.143860 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:47:19.528467 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.528438 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:47:19.624112 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz7r4\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624337 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624185 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624337 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624291 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624337 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624321 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624362 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624430 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624503 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624526 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy\") pod \"5f742c66-ea01-422c-9134-676adc457ded\" (UID: \"5f742c66-ea01-422c-9134-676adc457ded\") " Jul 27 22:47:19.624805 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624660 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:19.624865 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624810 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.624999 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.624976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:19.625238 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.625210 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:19.625463 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.625400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:47:19.625673 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.625647 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:19.626628 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.626602 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:47:19.626712 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.626630 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4" (OuterVolumeSpecName: "kube-api-access-mz7r4") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "kube-api-access-mz7r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:47:19.626760 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.626730 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:19.627278 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.627261 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f742c66-ea01-422c-9134-676adc457ded" (UID: "5f742c66-ea01-422c-9134-676adc457ded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:47:19.726023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.725988 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f742c66-ea01-422c-9134-676adc457ded-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726017 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f742c66-ea01-422c-9134-676adc457ded-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726028 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726023 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726036 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726327 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726044 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726327 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726052 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f742c66-ea01-422c-9134-676adc457ded-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726327 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726059 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7r4\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-kube-api-access-mz7r4\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:19.726327 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:19.726068 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f742c66-ea01-422c-9134-676adc457ded-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:20.395257 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.395224 2559 generic.go:356] "Generic (PLEG): container finished" podID="5f742c66-ea01-422c-9134-676adc457ded" containerID="e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16" exitCode=0 Jul 27 22:47:20.395432 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.395278 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" Jul 27 22:47:20.395432 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.395304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerDied","Data":"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16"} Jul 27 22:47:20.395432 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.395361 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6" event={"ID":"5f742c66-ea01-422c-9134-676adc457ded","Type":"ContainerDied","Data":"84b2f25c83cbdea8a9053987d613d71469620f965f99686e1f4b39c3ca123b4b"} Jul 27 22:47:20.395432 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.395383 2559 scope.go:117] "RemoveContainer" containerID="e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16" Jul 27 22:47:20.404537 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.404522 2559 scope.go:117] "RemoveContainer" containerID="bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648" Jul 27 22:47:20.413823 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.413805 2559 scope.go:117] "RemoveContainer" containerID="e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16" Jul 27 22:47:20.414008 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.413975 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:47:20.414139 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:47:20.414107 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16\": container with ID starting with e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16 not found: ID does not exist" containerID="e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16" Jul 27 22:47:20.414223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.414171 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16"} err="failed to get container status \"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16\": rpc error: code = NotFound desc = could not find container \"e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16\": container with ID starting with e08b19cd1ac89204663785c533cd3ac75f3f4fd32c80316bb43604829b056d16 not found: ID does not exist" Jul 27 22:47:20.414223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.414193 2559 scope.go:117] "RemoveContainer" containerID="bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648" Jul 27 22:47:20.414439 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:47:20.414414 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648\": container with ID starting with bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648 not found: ID does not exist" containerID="bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648" Jul 27 22:47:20.414439 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.414430 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648"} err="failed to get container status \"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648\": rpc error: code = NotFound desc = could not find container \"bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648\": container with ID starting with bf2b0c5fa87597eddd5e4f08cf1317696a6f4641e8d314405bf9eca9aa5fb648 not found: ID does not exist" Jul 27 22:47:20.417238 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:20.417219 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e47da1-openshift-default-549dc68b9b-wt7h6"] Jul 27 22:47:22.090267 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:22.090235 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f742c66-ea01-422c-9134-676adc457ded" path="/var/lib/kubelet/pods/5f742c66-ea01-422c-9134-676adc457ded/volumes" Jul 27 22:47:24.187862 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:24.187832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:24.288874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:24.288840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:25.227854 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:25.227823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:25.303818 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:25.303787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:25.368168 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:25.368115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:28.896988 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:28.896957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:29.711382 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:29.711350 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:30.093295 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:30.093212 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:36.281874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.281841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:36.349929 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.349898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:36.517685 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.517655 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:47:36.517890 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.517865 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" containerID="cri-o://b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d" gracePeriod=30 Jul 27 22:47:36.541799 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.541725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:36.743582 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:36.743545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:41.763710 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.763681 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:47:41.817103 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817022 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817109 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817179 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817280 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817239 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817274 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dtl7\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817307 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817390 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:41.817579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817461 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817579 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817499 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy\") pod \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\" (UID: \"d9a595e5-8e3d-47d2-bc9f-fb4e03852078\") " Jul 27 22:47:41.817736 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817556 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:41.817791 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817746 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:41.817791 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817777 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.817900 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817781 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:47:41.817900 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.817796 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.818194 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.818044 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:41.819605 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.819575 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:47:41.819718 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.819638 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7" (OuterVolumeSpecName: "kube-api-access-9dtl7") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "kube-api-access-9dtl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:47:41.819718 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.819661 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:47:41.819791 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.819756 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9a595e5-8e3d-47d2-bc9f-fb4e03852078" (UID: "d9a595e5-8e3d-47d2-bc9f-fb4e03852078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:47:41.918795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918757 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.918795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918785 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.918795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918796 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtl7\" (UniqueName: \"kubernetes.io/projected/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-kube-api-access-9dtl7\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.918795 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918806 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.919057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918814 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.919057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918822 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:41.919057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:41.918830 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a595e5-8e3d-47d2-bc9f-fb4e03852078-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:47:42.511199 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.511162 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerID="b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d" exitCode=0 Jul 27 22:47:42.511380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.511225 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" Jul 27 22:47:42.511380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.511246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerDied","Data":"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d"} Jul 27 22:47:42.511380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.511297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg" event={"ID":"d9a595e5-8e3d-47d2-bc9f-fb4e03852078","Type":"ContainerDied","Data":"f7cc134219b3bd91686a9e98246f26408cef7b62d6cfc7b733e305f31e8fc5a2"} Jul 27 22:47:42.511380 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.511318 2559 scope.go:117] "RemoveContainer" containerID="b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d" Jul 27 22:47:42.520303 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.520287 2559 scope.go:117] "RemoveContainer" containerID="d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47" Jul 27 22:47:42.528841 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.528823 2559 scope.go:117] "RemoveContainer" containerID="b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d" Jul 27 22:47:42.528957 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.528936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:47:42.529127 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:47:42.529109 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d\": container with ID starting with b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d not found: ID does not exist" containerID="b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d" Jul 27 22:47:42.529199 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.529139 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d"} err="failed to get container status \"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d\": rpc error: code = NotFound desc = could not find container \"b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d\": container with ID starting with b44caf4726e53f53da5ece5d4fb2c5d58f4ad7d7a1cdc718025a127493d41d0d not found: ID does not exist" Jul 27 22:47:42.529199 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.529178 2559 scope.go:117] "RemoveContainer" containerID="d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47" Jul 27 22:47:42.529415 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:47:42.529398 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47\": container with ID starting with d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47 not found: ID does not exist" containerID="d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47" Jul 27 22:47:42.529474 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.529423 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47"} err="failed to get container status \"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47\": rpc error: code = NotFound desc = could not find container \"d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47\": container with ID starting with d9d7a58f949a0f86cb1f1d3ffe038842f9d8035f0dbb3ba4bd0e96d65223dd47 not found: ID does not exist" Jul 27 22:47:42.533179 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:42.533144 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c5cbe3-openshift-default-6c768f6bd4-xmtdg"] Jul 27 22:47:44.084176 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:44.084118 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" path="/var/lib/kubelet/pods/d9a595e5-8e3d-47d2-bc9f-fb4e03852078/volumes" Jul 27 22:47:47.396413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:47.396377 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:47.495543 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:47.495499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:47.595647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:47.595601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:51.928825 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:51.928785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:52.299382 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:52.299291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:58.507320 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.507270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:58.575838 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.575802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:58.720223 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.720192 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:47:58.720418 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.720393 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" containerID="cri-o://3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d" gracePeriod=30 Jul 27 22:47:58.744342 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.744311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:58.950902 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:58.950867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:47:59.359864 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:47:59.359838 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995db75bc-j72mz_c1aeb91a-d48a-46ea-b5e0-aed2209630a3/authorino/0.log" Jul 27 22:48:03.971211 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:03.971191 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:48:04.104073 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.103993 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnzdc\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104073 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104073 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104062 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104200 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104246 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104270 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104316 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104364 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket\") pod \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\" (UID: \"0e2c91be-59e4-463d-9165-68d6d8c10fcd\") " Jul 27 22:48:04.104696 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104497 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:48:04.104696 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104674 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.104820 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104689 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:48:04.104875 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.104803 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:48:04.105200 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.105174 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:48:04.105290 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.105174 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:48:04.106429 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.106400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:48:04.106629 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.106610 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:48:04.106721 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.106708 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc" (OuterVolumeSpecName: "kube-api-access-mnzdc") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "kube-api-access-mnzdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:48:04.106832 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.106811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e2c91be-59e4-463d-9165-68d6d8c10fcd" (UID: "0e2c91be-59e4-463d-9165-68d6d8c10fcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:48:04.205531 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205496 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzdc\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-kube-api-access-mnzdc\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205531 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205523 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-envoy\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205531 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205534 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-podinfo\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205531 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205542 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-workload-certs\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205550 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-data\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205557 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istio-token\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205566 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e2c91be-59e4-463d-9165-68d6d8c10fcd-istiod-ca-cert\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.205787 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.205573 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e2c91be-59e4-463d-9165-68d6d8c10fcd-credential-socket\") on node \"ip-10-0-134-139.ec2.internal\" DevicePath \"\"" Jul 27 22:48:04.621665 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.621629 2559 generic.go:356] "Generic (PLEG): container finished" podID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerID="3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d" exitCode=0 Jul 27 22:48:04.621874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.621700 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" Jul 27 22:48:04.621874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.621709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerDied","Data":"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d"} Jul 27 22:48:04.621874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.621803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4" event={"ID":"0e2c91be-59e4-463d-9165-68d6d8c10fcd","Type":"ContainerDied","Data":"ee2c0bbea972f5873c900c05f2efa82528679ff8d3096f78a6657841fbae99a6"} Jul 27 22:48:04.621874 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.621835 2559 scope.go:117] "RemoveContainer" containerID="3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d" Jul 27 22:48:04.630926 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.630911 2559 scope.go:117] "RemoveContainer" containerID="38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d" Jul 27 22:48:04.639428 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.639413 2559 scope.go:117] "RemoveContainer" containerID="3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d" Jul 27 22:48:04.639710 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:48:04.639694 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d\": container with ID starting with 3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d not found: ID does not exist" containerID="3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d" Jul 27 22:48:04.639753 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.639722 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d"} err="failed to get container status \"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d\": rpc error: code = NotFound desc = could not find container \"3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d\": container with ID starting with 3baa31789d60174c4fe458a80d6568362470853b5dfe52b9c1d486c481ffbf5d not found: ID does not exist" Jul 27 22:48:04.639753 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.639738 2559 scope.go:117] "RemoveContainer" containerID="38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d" Jul 27 22:48:04.639974 ip-10-0-134-139 kubenswrapper[2559]: E0727 22:48:04.639957 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d\": container with ID starting with 38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d not found: ID does not exist" containerID="38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d" Jul 27 22:48:04.640013 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.639981 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d"} err="failed to get container status \"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d\": rpc error: code = NotFound desc = could not find container \"38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d\": container with ID starting with 38e7338fd82faa30c1d41722d8f95c5b49afafccf4086b1afd4fd52104190d7d not found: ID does not exist" Jul 27 22:48:04.643591 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.643569 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:48:04.649712 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.649689 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f36b41-openshift-default-5794ccfcbd-6dlm4"] Jul 27 22:48:04.937754 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:04.937667 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-548j6_9e4e80ad-ab46-421d-a00e-10ff6cca7e05/server/0.log" Jul 27 22:48:05.052648 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:05.052598 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5ln2z_25cb3262-3934-42da-add0-9d8d0ba9611e/manager/0.log" Jul 27 22:48:05.288942 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:05.288871 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fb7b654-q98t8_b42a6d2f-7d11-4d07-8632-99f8c32c406e/copy-manifests/0.log" Jul 27 22:48:05.315146 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:05.315118 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fb7b654-q98t8_b42a6d2f-7d11-4d07-8632-99f8c32c406e/manager/0.log" Jul 27 22:48:05.872413 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:05.872388 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753175-9fv9t_d1beba94-4b34-4fb4-b77a-ff48d2d02249/cleanup/0.log" Jul 27 22:48:05.977771 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:05.977743 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753190-q4brd_bd693fa0-3d5c-4cd3-896c-a66a8fab5118/cleanup/0.log" Jul 27 22:48:06.084191 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:06.084142 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" path="/var/lib/kubelet/pods/0e2c91be-59e4-463d-9165-68d6d8c10fcd/volumes" Jul 27 22:48:06.931612 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:06.931580 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/extract/0.log" Jul 27 22:48:06.937365 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:06.937344 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/util/0.log" Jul 27 22:48:06.943609 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:06.943572 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/pull/0.log" Jul 27 22:48:07.052976 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.052948 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/util/0.log" Jul 27 22:48:07.059050 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.059031 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/pull/0.log" Jul 27 22:48:07.065509 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.065467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/extract/0.log" Jul 27 22:48:07.169373 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.169349 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/util/0.log" Jul 27 22:48:07.174916 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.174897 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/pull/0.log" Jul 27 22:48:07.180436 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.180400 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/extract/0.log" Jul 27 22:48:07.284690 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.284603 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/util/0.log" Jul 27 22:48:07.290664 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.290643 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/pull/0.log" Jul 27 22:48:07.296680 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.296659 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/extract/0.log" Jul 27 22:48:07.409582 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:07.409552 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995db75bc-j72mz_c1aeb91a-d48a-46ea-b5e0-aed2209630a3/authorino/0.log" Jul 27 22:48:08.547698 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:08.547648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crxd6b_54bb52a8-a014-4832-a044-ebc8d6b2ee85/istio-proxy/0.log" Jul 27 22:48:08.891606 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:08.891575 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fcc8648dc-l6rzc_8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f/kube-auth-proxy/0.log" Jul 27 22:48:08.999702 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:08.999651 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/1.log" Jul 27 22:48:09.332057 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:09.331981 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fbddf666f-qzmvf_0c78a3d5-3758-4f24-9833-955c51dd5063/router/0.log" Jul 27 22:48:10.214256 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:10.214224 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cxpxh_10b66195-f639-4c34-a654-b1c07973da6f/main/0.log" Jul 27 22:48:15.374637 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j945x/must-gather-bv4wf"] Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374875 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374885 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374895 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374900 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374906 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374912 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374921 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374926 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374936 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374941 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374951 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.374956 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375000 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375007 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375012 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375019 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375025 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375030 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375035 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375081 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375077 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375086 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a595e5-8e3d-47d2-bc9f-fb4e03852078" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375092 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375098 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2c91be-59e4-463d-9165-68d6d8c10fcd" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375103 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375108 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375115 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375119 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375182 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbd7c06-6c92-4710-99a2-51d9f2a53ccd" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375190 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f742c66-ea01-422c-9134-676adc457ded" containerName="istio-proxy" Jul 27 22:48:15.375733 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.375196 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ebb3bb8-d2d9-402d-b69e-4cbb25f2aa36" containerName="istio-proxy" Jul 27 22:48:15.379585 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.379569 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.387978 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.387959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j945x\"/\"kube-root-ca.crt\"" Jul 27 22:48:15.388080 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.388021 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j945x\"/\"openshift-service-ca.crt\"" Jul 27 22:48:15.388376 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.388358 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j945x\"/\"default-dockercfg-94sd4\"" Jul 27 22:48:15.393892 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.393870 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j945x/must-gather-bv4wf"] Jul 27 22:48:15.401298 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.401274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:48:15.469752 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.469719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" limit=1024 Jul 27 22:48:15.491494 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.491465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/511fc220-dfeb-4344-a68e-ea2e3a080e1e-must-gather-output\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.491645 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.491505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wgb\" (UniqueName: \"kubernetes.io/projected/511fc220-dfeb-4344-a68e-ea2e3a080e1e-kube-api-access-58wgb\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.592235 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.592185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/511fc220-dfeb-4344-a68e-ea2e3a080e1e-must-gather-output\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.592420 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.592254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wgb\" (UniqueName: \"kubernetes.io/projected/511fc220-dfeb-4344-a68e-ea2e3a080e1e-kube-api-access-58wgb\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.592520 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.592501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/511fc220-dfeb-4344-a68e-ea2e3a080e1e-must-gather-output\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.600556 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.600525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wgb\" (UniqueName: \"kubernetes.io/projected/511fc220-dfeb-4344-a68e-ea2e3a080e1e-kube-api-access-58wgb\") pod \"must-gather-bv4wf\" (UID: \"511fc220-dfeb-4344-a68e-ea2e3a080e1e\") " pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.688301 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.688218 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j945x/must-gather-bv4wf" Jul 27 22:48:15.819238 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.819215 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j945x/must-gather-bv4wf"] Jul 27 22:48:15.821124 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:48:15.821096 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511fc220_dfeb_4344_a68e_ea2e3a080e1e.slice/crio-af5dd8202c1c438f5881bccef89e05b42307e5982283eaf6c9c2c3335707199e WatchSource:0}: Error finding container af5dd8202c1c438f5881bccef89e05b42307e5982283eaf6c9c2c3335707199e: Status 404 returned error can't find the container with id af5dd8202c1c438f5881bccef89e05b42307e5982283eaf6c9c2c3335707199e Jul 27 22:48:15.822893 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:15.822877 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:48:16.687369 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:16.687333 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j945x/must-gather-bv4wf" event={"ID":"511fc220-dfeb-4344-a68e-ea2e3a080e1e","Type":"ContainerStarted","Data":"af5dd8202c1c438f5881bccef89e05b42307e5982283eaf6c9c2c3335707199e"} Jul 27 22:48:17.695075 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:17.695040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j945x/must-gather-bv4wf" event={"ID":"511fc220-dfeb-4344-a68e-ea2e3a080e1e","Type":"ContainerStarted","Data":"9fa8c58c93bb2d44b8106bd9965c3a3e67c544617c313e11b0f160ea1609e388"} Jul 27 22:48:17.695490 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:17.695090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j945x/must-gather-bv4wf" event={"ID":"511fc220-dfeb-4344-a68e-ea2e3a080e1e","Type":"ContainerStarted","Data":"54d1428cf4e722e258df1b5b83622f40fb33e7414b92038f8c918474afa395a6"} Jul 27 22:48:17.713347 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:17.713286 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j945x/must-gather-bv4wf" podStartSLOduration=1.658208959 podStartE2EDuration="2.71326424s" totalImagesPullingTime="1.055055281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:48:17.710874523 +0000 UTC m=+3218.091946446" watchObservedRunningTime="2026-07-27 22:48:17.71326424 +0000 UTC m=+3218.094336163" Jul 27 22:48:18.473016 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:18.472991 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-grl2x_8a851d07-33d7-4bcc-99e2-eb277ac875d4/global-pull-secret-syncer/0.log" Jul 27 22:48:18.665397 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:18.665370 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdz5l_c3df729f-a496-4eab-bde4-60eb6e6cffe7/konnectivity-agent/0.log" Jul 27 22:48:18.776529 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:18.776450 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-139.ec2.internal_103daea59f1aca008ae67124379b3014/haproxy/0.log" Jul 27 22:48:23.072262 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.072178 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/extract/0.log" Jul 27 22:48:23.098500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.098470 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/util/0.log" Jul 27 22:48:23.128691 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.128330 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pcpws_0201d9de-dffb-41a7-b193-a62458ab7f55/pull/0.log" Jul 27 22:48:23.163950 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.163792 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/extract/0.log" Jul 27 22:48:23.189855 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.189763 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/util/0.log" Jul 27 22:48:23.215227 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.215125 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0ngcgf_3b626c95-aa0b-4da3-930a-df485f8e541d/pull/0.log" Jul 27 22:48:23.262088 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.262011 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/extract/0.log" Jul 27 22:48:23.288383 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.288356 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/util/0.log" Jul 27 22:48:23.315586 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.315560 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735sjnp_f5f06467-3337-483e-88c0-6f3513060a38/pull/0.log" Jul 27 22:48:23.358094 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.357995 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/extract/0.log" Jul 27 22:48:23.389207 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.386629 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/util/0.log" Jul 27 22:48:23.415107 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.415079 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t4q9v_e6363ef5-506f-4f64-a544-fd95ac4653e2/pull/0.log" Jul 27 22:48:23.865172 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:23.863222 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995db75bc-j72mz_c1aeb91a-d48a-46ea-b5e0-aed2209630a3/authorino/0.log" Jul 27 22:48:26.332299 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:26.332265 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/1.log" Jul 27 22:48:26.409972 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:26.409821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-7gtpr_f988c022-2428-446d-9ae1-71c26ae9bbfb/cluster-monitoring-operator/0.log" Jul 27 22:48:26.654084 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:26.654039 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7n6c_dc2b703a-a49d-4b0e-a35e-ac6239d4ae39/node-exporter/0.log" Jul 27 22:48:26.678119 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:26.678047 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7n6c_dc2b703a-a49d-4b0e-a35e-ac6239d4ae39/kube-rbac-proxy/0.log" Jul 27 22:48:26.702461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:26.702431 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7n6c_dc2b703a-a49d-4b0e-a35e-ac6239d4ae39/init-textfile/0.log" Jul 27 22:48:28.571725 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.571688 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr"] Jul 27 22:48:28.577755 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.577730 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.587979 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.587836 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr"] Jul 27 22:48:28.734304 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.734268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-sys\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.734506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.734319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-podres\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.734506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.734380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-lib-modules\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.734506 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.734473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-proc\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.734662 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.734543 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvw2p\" (UniqueName: \"kubernetes.io/projected/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-kube-api-access-dvw2p\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835426 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-sys\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835615 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-podres\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-lib-modules\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835741 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-proc\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835850 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvw2p\" (UniqueName: \"kubernetes.io/projected/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-kube-api-access-dvw2p\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-podres\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.835903 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-proc\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.836002 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.835908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-lib-modules\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.836130 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.836110 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-sys\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.847391 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.847335 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvw2p\" (UniqueName: \"kubernetes.io/projected/259bf2f6-d456-4e3a-b7fa-8edaa0465acc-kube-api-access-dvw2p\") pod \"perf-node-gather-daemonset-fndbr\" (UID: \"259bf2f6-d456-4e3a-b7fa-8edaa0465acc\") " pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:28.893232 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:28.893194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:29.084942 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:29.083431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr"] Jul 27 22:48:29.086250 ip-10-0-134-139 kubenswrapper[2559]: W0727 22:48:29.086145 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod259bf2f6_d456_4e3a_b7fa_8edaa0465acc.slice/crio-5bf8e1877f05d74e49f05c916830524be0cf5eb1abbf401224015a6d8b732173 WatchSource:0}: Error finding container 5bf8e1877f05d74e49f05c916830524be0cf5eb1abbf401224015a6d8b732173: Status 404 returned error can't find the container with id 5bf8e1877f05d74e49f05c916830524be0cf5eb1abbf401224015a6d8b732173 Jul 27 22:48:29.784020 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:29.783975 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" event={"ID":"259bf2f6-d456-4e3a-b7fa-8edaa0465acc","Type":"ContainerStarted","Data":"a51eb1d9b59a6baa5486529c3b05310421762180def54ace8bc6a18fb4c1671d"} Jul 27 22:48:29.784546 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:29.784049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" event={"ID":"259bf2f6-d456-4e3a-b7fa-8edaa0465acc","Type":"ContainerStarted","Data":"5bf8e1877f05d74e49f05c916830524be0cf5eb1abbf401224015a6d8b732173"} Jul 27 22:48:29.785248 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:29.785222 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:29.807317 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:29.807265 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" podStartSLOduration=1.80724741 podStartE2EDuration="1.80724741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:48:29.805965436 +0000 UTC m=+3230.187037361" watchObservedRunningTime="2026-07-27 22:48:29.80724741 +0000 UTC m=+3230.188319339" Jul 27 22:48:31.321050 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:31.321019 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzf7h_a6f1f903-b9f0-468c-9a33-4294fa33d347/dns/0.log" Jul 27 22:48:31.345934 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:31.345887 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzf7h_a6f1f903-b9f0-468c-9a33-4294fa33d347/kube-rbac-proxy/0.log" Jul 27 22:48:31.421716 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:31.421687 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzr5j_5343a980-96ea-4633-9a06-5511934c207f/dns-node-resolver/0.log" Jul 27 22:48:32.091910 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:32.091872 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5nx9_8f3b1d48-c616-4c79-8405-fe3c3f5cb03f/node-ca/0.log" Jul 27 22:48:33.011007 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:33.010974 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crxd6b_54bb52a8-a014-4832-a044-ebc8d6b2ee85/istio-proxy/0.log" Jul 27 22:48:33.334532 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:33.334461 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fcc8648dc-l6rzc_8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f/kube-auth-proxy/0.log" Jul 27 22:48:33.363688 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:33.363638 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/0.log" Jul 27 22:48:33.400411 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:33.400362 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/1.log" Jul 27 22:48:33.504970 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:33.504942 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fbddf666f-qzmvf_0c78a3d5-3758-4f24-9833-955c51dd5063/router/0.log" Jul 27 22:48:34.325609 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:34.325585 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9kz5_e7e538e7-d2a1-4e19-8fd3-f8a2d05fdbd6/serve-healthcheck-canary/0.log" Jul 27 22:48:34.878616 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:34.878582 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5n9wf_ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8/kube-rbac-proxy/0.log" Jul 27 22:48:34.902437 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:34.902413 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5n9wf_ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8/exporter/0.log" Jul 27 22:48:34.925568 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:34.925542 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5n9wf_ae19c50f-c24b-4bb8-a8ec-cf8bc1ce14e8/extractor/0.log" Jul 27 22:48:36.808461 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:36.808432 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j945x/perf-node-gather-daemonset-fndbr" Jul 27 22:48:37.791699 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:37.791653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-548j6_9e4e80ad-ab46-421d-a00e-10ff6cca7e05/server/0.log" Jul 27 22:48:37.888458 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:37.888412 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5ln2z_25cb3262-3934-42da-add0-9d8d0ba9611e/manager/0.log" Jul 27 22:48:38.089491 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.089420 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fb7b654-q98t8_b42a6d2f-7d11-4d07-8632-99f8c32c406e/manager/0.log" Jul 27 22:48:38.108630 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.108600 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fb7b654-q98t8_b42a6d2f-7d11-4d07-8632-99f8c32c406e/copy-manifests/0.log" Jul 27 22:48:38.522485 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.522450 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crxd6b_54bb52a8-a014-4832-a044-ebc8d6b2ee85/istio-proxy/0.log" Jul 27 22:48:38.823074 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.823050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fcc8648dc-l6rzc_8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f/kube-auth-proxy/0.log" Jul 27 22:48:38.859968 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.859908 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/0.log" Jul 27 22:48:38.897700 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.897665 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/1.log" Jul 27 22:48:38.974340 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:38.974300 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fbddf666f-qzmvf_0c78a3d5-3758-4f24-9833-955c51dd5063/router/0.log" Jul 27 22:48:39.598638 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:39.598596 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8444c4f6f9-9zrk5_6d39fa14-2cf3-4fb3-9474-16c4992b379c/manager/0.log" Jul 27 22:48:40.481993 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:40.481966 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crxd6b_54bb52a8-a014-4832-a044-ebc8d6b2ee85/istio-proxy/0.log" Jul 27 22:48:40.772907 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:40.772813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fcc8648dc-l6rzc_8ce754b3-cb82-48f5-9d5a-2e6e53a92d1f/kube-auth-proxy/0.log" Jul 27 22:48:40.793905 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:40.793873 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/0.log" Jul 27 22:48:40.830842 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:40.830811 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rg9d4_96892165-bb45-4963-98fd-084a637f2b7d/istio-proxy/1.log" Jul 27 22:48:40.919407 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:40.919375 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fbddf666f-qzmvf_0c78a3d5-3758-4f24-9833-955c51dd5063/router/0.log" Jul 27 22:48:46.200886 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.200857 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/kube-multus-additional-cni-plugins/0.log" Jul 27 22:48:46.224561 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.224533 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/egress-router-binary-copy/0.log" Jul 27 22:48:46.247014 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.246985 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/cni-plugins/0.log" Jul 27 22:48:46.271238 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.271211 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/bond-cni-plugin/0.log" Jul 27 22:48:46.297224 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.297193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/routeoverride-cni/0.log" Jul 27 22:48:46.321647 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.321620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/whereabouts-cni-bincopy/0.log" Jul 27 22:48:46.345697 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.345674 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jb6g_41bb9b0c-eb9b-45df-8601-3ef7e1563049/whereabouts-cni/0.log" Jul 27 22:48:46.555478 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.555398 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kr2lz_1f243f54-1d60-4ff0-919f-86406da12c9d/kube-multus/0.log" Jul 27 22:48:46.750615 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.750586 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzk78_ff07cd87-c25a-4f0c-a5e8-4b172891c63b/network-metrics-daemon/0.log" Jul 27 22:48:46.771500 ip-10-0-134-139 kubenswrapper[2559]: I0727 22:48:46.771476 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzk78_ff07cd87-c25a-4f0c-a5e8-4b172891c63b/kube-rbac-proxy/0.log"