Jul 29 06:20:37.139578 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 06:20:37.139594 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 06:20:37.139603 ip-10-0-131-31 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 06:20:37.140015 ip-10-0-131-31 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 06:20:46.581248 ip-10-0-131-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 06:20:46.581260 ip-10-0-131-31 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e51801d021d64177aafddc45b228eced -- Jul 29 06:22:35.366650 ip-10-0-131-31 systemd[1]: Starting Kubernetes Kubelet... Jul 29 06:22:35.702829 ip-10-0-131-31 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 06:22:35.702829 ip-10-0-131-31 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 06:22:35.702829 ip-10-0-131-31 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 06:22:35.702829 ip-10-0-131-31 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 06:22:35.702829 ip-10-0-131-31 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 06:22:35.704538 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.704427 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 06:22:35.708785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708770 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.708785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708785 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708789 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708792 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708796 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708799 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708802 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708804 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708808 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708810 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708813 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708815 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708819 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708822 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708824 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708827 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708829 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708832 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708834 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708837 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708854 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.708852 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708857 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708860 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708862 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708865 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708868 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708871 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708873 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708876 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708878 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708881 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708883 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708885 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708888 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708890 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708893 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708895 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708898 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708900 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708902 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.709352 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708905 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708907 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708910 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708913 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708915 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708918 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708920 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708923 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708925 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708927 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708930 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708932 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708935 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708943 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708946 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708949 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708951 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708954 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708958 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708963 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.709823 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708966 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708968 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708971 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708973 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708976 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708978 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708981 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708983 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708987 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708991 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708993 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708996 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.708998 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709001 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709003 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709007 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709010 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709012 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709015 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.710313 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709019 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709022 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709025 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709028 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709031 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709033 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709036 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709043 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709046 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709049 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709052 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709054 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709056 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709059 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709062 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709064 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709066 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709069 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709071 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.710785 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.709074 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710918 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710927 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710930 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710933 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710936 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710938 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710942 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710944 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710947 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710949 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710951 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710954 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710957 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710959 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710962 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710964 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710966 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710969 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710972 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.711241 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710974 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710977 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710980 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710982 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710984 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710987 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710990 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710992 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710994 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.710997 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711000 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711002 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711004 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711007 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711010 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711012 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711015 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711018 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711020 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711023 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.711777 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711025 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711028 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711030 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711032 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711035 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711037 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711039 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711042 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711046 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711048 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711051 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711054 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711057 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711059 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711062 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711064 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711067 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711069 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711072 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.712269 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711074 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711077 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711081 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711084 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711087 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711090 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711093 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711096 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711099 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711102 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711104 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711108 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711111 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711113 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711116 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711118 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711120 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711122 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711125 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711127 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.712748 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711130 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711132 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711134 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711137 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711139 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711142 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711144 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711147 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711150 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711152 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711155 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711157 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711159 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711162 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711165 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711169 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711172 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711174 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711177 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.713229 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711180 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711183 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711548 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711553 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711556 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711559 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711562 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711566 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711569 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711572 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711574 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711577 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711580 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711583 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711587 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711590 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711593 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711596 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711599 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711602 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.713702 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711604 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711608 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711610 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711613 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711616 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711618 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711621 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711623 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711625 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711628 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711630 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711633 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711636 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711638 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711641 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711643 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711646 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711648 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711652 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711654 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.714178 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711656 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711658 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711663 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711666 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711668 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711671 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711674 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711676 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711679 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711682 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711684 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711687 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711689 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711692 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711695 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711697 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711700 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711702 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711705 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.714664 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711708 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711710 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711712 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711715 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711717 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711720 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711723 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711725 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711727 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711730 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711732 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711734 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711737 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711739 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711742 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711745 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711747 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711750 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711753 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711755 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.715122 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711758 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711760 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711763 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711765 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711768 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711770 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711773 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711775 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711778 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711781 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711783 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711786 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711788 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711791 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711794 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711796 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711798 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711801 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711803 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.715608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711806 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711808 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.711812 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711885 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711892 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711898 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711902 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711906 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711909 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711913 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711917 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711920 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711923 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711926 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711929 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711933 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711935 2556 flags.go:64] FLAG: --cgroup-root="" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711938 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711941 2556 flags.go:64] FLAG: --client-ca-file="" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711944 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711947 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711951 2556 flags.go:64] FLAG: --cluster-domain="" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711955 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 06:22:35.716065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711958 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711961 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711964 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711968 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711971 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711974 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711977 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711980 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711983 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711985 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711989 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711992 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711995 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.711998 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712001 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712003 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712007 2556 flags.go:64] FLAG: --enable-server="true" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712009 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712013 2556 flags.go:64] FLAG: --event-burst="100" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712016 2556 flags.go:64] FLAG: --event-qps="50" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712019 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712022 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712025 2556 flags.go:64] FLAG: --eviction-hard="" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712029 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 06:22:35.716637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712031 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712034 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712037 2556 flags.go:64] FLAG: --eviction-soft="" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712042 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712045 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712048 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712051 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712054 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712057 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712060 2556 flags.go:64] FLAG: --feature-gates="" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712063 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712066 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712069 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712072 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712075 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712078 2556 flags.go:64] FLAG: --help="false" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712080 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-131-31.ec2.internal" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712083 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712086 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712088 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712092 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712095 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712097 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 06:22:35.717277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712100 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712103 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712106 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712108 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712111 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712114 2556 flags.go:64] FLAG: --kube-reserved="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712117 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712119 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712122 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712125 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712127 2556 flags.go:64] FLAG: --lock-file="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712130 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712134 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712136 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712141 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712144 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712147 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712150 2556 flags.go:64] FLAG: --logging-format="text" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712156 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712159 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712162 2556 flags.go:64] FLAG: --manifest-url="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712164 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712169 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712171 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712175 2556 flags.go:64] FLAG: --max-pods="110" Jul 29 06:22:35.717886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712179 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712182 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712184 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712187 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712190 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712193 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712196 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712203 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712210 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712213 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712216 2556 flags.go:64] FLAG: --pod-cidr="" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712218 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712224 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712226 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712229 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712232 2556 flags.go:64] FLAG: --port="10250" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712235 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712238 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-023a5c83d9bcc490b" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712241 2556 flags.go:64] FLAG: --qos-reserved="" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712243 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712247 2556 flags.go:64] FLAG: --register-node="true" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712250 2556 flags.go:64] FLAG: --register-with-taints="" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712253 2556 flags.go:64] FLAG: --registry-burst="10" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712256 2556 flags.go:64] FLAG: --registry-qps="5" Jul 29 06:22:35.718498 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712259 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712261 2556 flags.go:64] FLAG: --reserved-memory="" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712265 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712268 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712271 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712274 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712277 2556 flags.go:64] FLAG: --runonce="false" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712280 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712283 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712286 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712288 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712291 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712294 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712297 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712300 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712302 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712306 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712309 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712312 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712315 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712317 2556 flags.go:64] FLAG: --system-cgroups="" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712320 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712325 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712327 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712330 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 06:22:35.719071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712334 2556 flags.go:64] FLAG: --tls-min-version="" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712337 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712339 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712343 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712346 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712349 2556 flags.go:64] FLAG: --v="2" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712352 2556 flags.go:64] FLAG: --version="false" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712356 2556 flags.go:64] FLAG: --vmodule="" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712360 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.712363 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712485 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712489 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712492 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712496 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712499 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712503 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712506 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712509 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712512 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712514 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712516 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.719792 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712519 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712521 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712525 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712528 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712530 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712533 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712535 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712538 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712540 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712543 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712545 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712547 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712550 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712552 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712556 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712558 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712561 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712563 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712565 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712568 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.720301 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712570 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712573 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712576 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712578 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712581 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712583 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712585 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712588 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712591 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712593 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712596 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712598 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712601 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712603 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712607 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712609 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712612 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712614 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712616 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712619 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.720824 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712622 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712624 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712626 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712629 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712631 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712634 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712638 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712641 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712643 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712646 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712648 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712651 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712653 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712656 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712658 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712661 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712663 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712666 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712669 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712671 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.721588 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712673 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712676 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712678 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712681 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712683 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712685 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712689 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712692 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712694 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712698 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712701 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712704 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712707 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712710 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712712 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712714 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712717 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712719 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.722203 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712722 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712726 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712730 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712733 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712735 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712738 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712741 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712743 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712746 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.712749 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.713308 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.722450 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 06:22:35.722671 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.722463 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 06:22:35.723195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.723184 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 06:22:35.723226 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.723199 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 06:22:35.723262 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723253 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.723262 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723261 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723264 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723267 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723270 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723273 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723275 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723278 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723281 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723284 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723286 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723289 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723292 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723295 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723302 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723305 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723308 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723310 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723313 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723315 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723317 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.723315 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723320 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723323 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723326 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723329 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723332 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723335 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723337 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723340 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723342 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723345 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723347 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723350 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723352 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723355 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723357 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723360 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723362 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723365 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723369 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.723830 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723372 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723393 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723397 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723400 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723402 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723405 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723409 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723411 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723414 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723416 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723419 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723426 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723429 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723431 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723434 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723438 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723440 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723443 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723446 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723448 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.724285 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723451 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723453 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723456 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723458 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723461 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723463 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723465 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723468 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723470 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723473 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723476 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723479 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723482 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723484 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723487 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723489 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723492 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723494 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723496 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.724812 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723499 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723502 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723504 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723507 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723509 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723511 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723514 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723517 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723519 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723522 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723524 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723527 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723529 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723532 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723535 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723537 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723539 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723542 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723544 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.725255 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723546 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.723552 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723673 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723678 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723681 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723684 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723687 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723690 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723692 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723695 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723698 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723700 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723703 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723706 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723709 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 06:22:35.725734 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723712 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723714 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723717 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723719 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723722 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723724 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723726 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723729 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723731 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723734 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723736 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723739 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723741 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723743 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723746 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723748 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723751 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723753 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723756 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723758 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 06:22:35.726095 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723760 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723763 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723765 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723768 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723770 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723772 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723775 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723777 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723779 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723782 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723785 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723787 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723791 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723795 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723798 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723801 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723804 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723806 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723809 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 06:22:35.726592 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723812 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723815 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723818 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723821 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723825 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723828 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723831 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723834 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723837 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723839 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723843 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723846 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723848 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723851 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723854 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723857 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723860 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723862 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723865 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 06:22:35.727053 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723867 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723870 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723872 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723874 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723877 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723880 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723882 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723885 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723887 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723890 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723892 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723895 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723897 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723900 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723902 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723904 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723907 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723909 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723911 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723914 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 06:22:35.727526 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723916 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723918 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723921 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723925 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723927 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723930 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723932 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:35.723935 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.723939 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.724038 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 06:22:35.727992 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.727475 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 06:22:35.728807 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.728795 2556 server.go:1019] "Starting client certificate rotation" Jul 29 06:22:35.728921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.728903 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 06:22:35.728953 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.728948 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 06:22:35.731157 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.731144 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 06:22:35.734248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.734233 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 06:22:35.742935 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.742910 2556 log.go:25] "Validated CRI v1 runtime API" Jul 29 06:22:35.746717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.746691 2556 log.go:25] "Validated CRI v1 image API" Jul 29 06:22:35.747485 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.747470 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 06:22:35.749579 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.749562 2556 fs.go:135] Filesystem UUIDs: map[6eb8def0-a9c1-4d5d-8ac5-9a13f5733320:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89bfa75c-c426-4f75-acd4-74fffa1154f8:/dev/nvme0n1p3] Jul 29 06:22:35.749628 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.749580 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 06:22:35.754912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.754810 2556 manager.go:216] Machine: {Timestamp:2026-07-29 06:22:35.753702365 +0000 UTC m=+0.287781929 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089746 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27740b248821474398f3643639d000 SystemUUID:ec27740b-2488-2147-4398-f3643639d000 BootID:e51801d0-21d6-4177-aafd-dc45b228eced Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:96:6f:a6:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:96:6f:a6:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:ea:30:2d:f3:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 06:22:35.754912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.754905 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 06:22:35.755021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.754976 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 06:22:35.757874 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.757851 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 06:22:35.758028 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.757883 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 06:22:35.758994 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.758981 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 06:22:35.759061 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.758996 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 06:22:35.759061 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.759004 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 06:22:35.759657 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.759645 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 06:22:35.760840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.760829 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 06:22:35.761916 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.761901 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 06:22:35.763677 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763665 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 29 06:22:35.763816 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763806 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 06:22:35.763855 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763836 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 29 06:22:35.763890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763857 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 06:22:35.763890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763869 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 06:22:35.763890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763882 2556 kubelet.go:398] "Adding apiserver pod source" Jul 29 06:22:35.764015 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.763896 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 06:22:35.764605 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.764585 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 06:22:35.764688 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.764615 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 06:22:35.767178 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.767137 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 06:22:35.768807 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.768787 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 06:22:35.768895 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.768828 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 06:22:35.769329 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769316 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769336 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769345 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769354 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769366 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769395 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769408 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 06:22:35.769414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769416 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 06:22:35.769634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769426 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 06:22:35.769634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769433 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 06:22:35.769634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769445 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 06:22:35.769634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.769456 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 06:22:35.770855 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.770842 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 06:22:35.770908 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.770858 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 06:22:35.773902 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.773886 2556 server.go:1268] "Started kubelet" Jul 29 06:22:35.773991 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.773967 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 06:22:35.774222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.774157 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 06:22:35.774295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.774234 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 06:22:35.774405 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.774354 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 06:22:35.774742 ip-10-0-131-31 systemd[1]: Started Kubernetes Kubelet. Jul 29 06:22:35.774968 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.774935 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 06:22:35.774968 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.774934 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 06:22:35.775740 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.775720 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 06:22:35.775839 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.775825 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 29 06:22:35.778803 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.778782 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 06:22:35.779388 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.778563 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-31.ec2.internal.18c6ae7d5e381082 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-31.ec2.internal,UID:ip-10-0-131-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-31.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.773857922 +0000 UTC m=+0.307937487,LastTimestamp:2026-07-29 06:22:35.773857922 +0000 UTC m=+0.307937487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-31.ec2.internal,}" Jul 29 06:22:35.781710 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.781693 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 06:22:35.781710 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.781708 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 06:22:35.781895 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.781876 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:35.782247 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782230 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 06:22:35.782247 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782230 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 06:22:35.782390 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782277 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 06:22:35.782451 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782443 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 29 06:22:35.782451 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782451 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 29 06:22:35.782552 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782538 2556 factory.go:55] Registering systemd factory Jul 29 06:22:35.782602 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782558 2556 factory.go:223] Registration of the systemd container factory successfully Jul 29 06:22:35.782749 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782739 2556 factory.go:153] Registering CRI-O factory Jul 29 06:22:35.782749 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782749 2556 factory.go:223] Registration of the crio container factory successfully Jul 29 06:22:35.782849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782792 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 06:22:35.782849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782810 2556 factory.go:103] Registering Raw factory Jul 29 06:22:35.782849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.782821 2556 manager.go:1196] Started watching for new ooms in manager Jul 29 06:22:35.783405 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.783368 2556 manager.go:349] Starting recovery of all containers Jul 29 06:22:35.784113 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.784092 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 06:22:35.794071 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.794036 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 06:22:35.794165 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.794086 2556 manager.go:354] Recovery completed Jul 29 06:22:35.795986 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.795966 2556 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 29 06:22:35.798978 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.798879 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:35.801293 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801268 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:35.801360 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801310 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:35.801360 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801324 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:35.801879 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801851 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 06:22:35.801879 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801868 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 06:22:35.801879 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801870 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:35.802057 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.801887 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 29 06:22:35.802502 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.802489 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:35.802551 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.802513 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:35.802551 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.802522 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:35.803113 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.803100 2556 policy_none.go:49] "None policy: Start" Jul 29 06:22:35.803180 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.803120 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 06:22:35.803180 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.803134 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 06:22:35.803627 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.803610 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7wx7" Jul 29 06:22:35.804450 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.804439 2556 policy_none.go:47] "Start" Jul 29 06:22:35.809413 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.809396 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7wx7" Jul 29 06:22:35.811451 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.811362 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-31.ec2.internal.18c6ae7d5fdab5a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-31.ec2.internal,UID:ip-10-0-131-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-31.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.801294244 +0000 UTC m=+0.335373813,LastTimestamp:2026-07-29 06:22:35.801294244 +0000 UTC m=+0.335373813,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-31.ec2.internal,}" Jul 29 06:22:35.843943 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.843929 2556 manager.go:337] "Starting Device Plugin manager" Jul 29 06:22:35.844036 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.843966 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 06:22:35.844036 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.843975 2556 server.go:87] "Starting device plugin registration server" Jul 29 06:22:35.844210 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.844196 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 06:22:35.844259 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.844212 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 06:22:35.844301 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.844265 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 06:22:35.844349 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.844336 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 06:22:35.844464 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.844350 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 06:22:35.844882 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.844852 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 06:22:35.844959 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.844909 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:35.919973 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.919940 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 06:22:35.921301 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.921285 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 06:22:35.921399 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.921329 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 06:22:35.921399 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.921355 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 06:22:35.921509 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.921413 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 06:22:35.923518 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.923497 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:35.945355 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.945327 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:35.946316 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.946295 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:35.946462 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.946334 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:35.946462 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.946350 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:35.946462 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.946403 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-31.ec2.internal" Jul 29 06:22:35.954086 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:35.954040 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-31.ec2.internal" Jul 29 06:22:35.954086 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.954064 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-31.ec2.internal\": node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:35.976555 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:35.976535 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.022479 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.022455 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal"] Jul 29 06:22:36.022540 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.022524 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:36.024308 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.024286 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:36.024409 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.024320 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:36.024409 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.024330 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:36.025480 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.025468 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:36.025596 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.025581 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.025661 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.025615 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:36.026204 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026186 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:36.026305 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026212 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:36.026305 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026226 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:36.026305 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026286 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:36.026305 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026305 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:36.026494 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.026314 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:36.027331 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.027316 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.027427 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.027345 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 06:22:36.028047 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.028032 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientMemory" Jul 29 06:22:36.028122 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.028063 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 06:22:36.028122 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.028075 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeHasSufficientPID" Jul 29 06:22:36.048033 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.048010 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-31.ec2.internal\" not found" node="ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.052231 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.052215 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-31.ec2.internal\" not found" node="ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.077184 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.077167 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.083487 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.083470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.083539 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.083492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.083539 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.083508 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/664c079835b1aca0098a8fa9d9f98d07-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"664c079835b1aca0098a8fa9d9f98d07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.177625 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.177593 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.183890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.183971 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183902 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.183971 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/664c079835b1aca0098a8fa9d9f98d07-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"664c079835b1aca0098a8fa9d9f98d07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.183971 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.184090 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183978 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c99cada63a53cc0230aa733d5f30bd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal\" (UID: \"9c99cada63a53cc0230aa733d5f30bd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.184090 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.183995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/664c079835b1aca0098a8fa9d9f98d07-config\") pod \"kube-apiserver-proxy-ip-10-0-131-31.ec2.internal\" (UID: \"664c079835b1aca0098a8fa9d9f98d07\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.278401 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.278291 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.349833 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.349805 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.355164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.355147 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.379064 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.379044 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.479656 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.479616 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.580167 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.580098 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.598025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.598001 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:36.683944 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.683904 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.729160 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.729130 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 06:22:36.729651 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.729312 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 06:22:36.729651 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.729347 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 06:22:36.784654 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:36.784626 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-31.ec2.internal\" not found" Jul 29 06:22:36.813286 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.813251 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 06:17:36 +0000 UTC" deadline="2028-01-06 12:01:16.651156928 +0000 UTC" Jul 29 06:22:36.813286 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.813283 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12629h38m39.837879993s" Jul 29 06:22:36.826862 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.826841 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:36.882949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.882882 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.893396 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.893375 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 06:22:36.894356 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.894340 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" Jul 29 06:22:36.906421 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.906394 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 06:22:36.986554 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:36.986502 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664c079835b1aca0098a8fa9d9f98d07.slice/crio-6c8d760a9f9944884825c2da4fd4451d6534d2c9db051a46f4dbc232662bb180 WatchSource:0}: Error finding container 6c8d760a9f9944884825c2da4fd4451d6534d2c9db051a46f4dbc232662bb180: Status 404 returned error can't find the container with id 6c8d760a9f9944884825c2da4fd4451d6534d2c9db051a46f4dbc232662bb180 Jul 29 06:22:36.986836 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:36.986808 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c99cada63a53cc0230aa733d5f30bd1.slice/crio-4c32467037b9432461841cd0d38ba868bf257086f16d92e3db47a8f65a082792 WatchSource:0}: Error finding container 4c32467037b9432461841cd0d38ba868bf257086f16d92e3db47a8f65a082792: Status 404 returned error can't find the container with id 4c32467037b9432461841cd0d38ba868bf257086f16d92e3db47a8f65a082792 Jul 29 06:22:36.991753 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:36.991736 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 06:22:37.023136 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.023117 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:37.766424 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.766360 2556 apiserver.go:52] "Watching apiserver" Jul 29 06:22:37.776391 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.776355 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 06:22:37.776950 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.776923 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-tg47d","openshift-image-registry/node-ca-6hn27","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal","openshift-multus/multus-789b6","openshift-multus/multus-additional-cni-plugins-9r4bn","openshift-network-diagnostics/network-check-target-5vfcd","kube-system/global-pull-secret-syncer-dg9h9","kube-system/konnectivity-agent-c2ghx","kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw","openshift-multus/network-metrics-daemon-dqvg9","openshift-network-operator/iptables-alerter-nw6mw","openshift-ovn-kubernetes/ovnkube-node-ksftp","openshift-cluster-node-tuning-operator/tuned-46lfr"] Jul 29 06:22:37.778493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.778462 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.780034 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.780008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.781794 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.782011 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.782329 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9zjjc\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.782544 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.782791 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.783840 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.784042 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltgbm\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.784277 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.784488 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.784819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-789b6" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.785794 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.785800 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.785980 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.786039 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kx6w7\"" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.786431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.786491 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 06:22:37.787182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.786533 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.788013 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.787680 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.788048 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788032 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4hsg4\"" Jul 29 06:22:37.788261 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788244 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 06:22:37.788912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788592 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 06:22:37.788912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788738 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 06:22:37.788912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788754 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.788912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.788889 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.789532 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.789510 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 06:22:37.790407 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790010 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wvrr2\"" Jul 29 06:22:37.790407 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790085 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-25nsr\"" Jul 29 06:22:37.790407 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790230 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 06:22:37.790407 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790287 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 06:22:37.790635 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790488 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.790635 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790550 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.790730 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790665 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 06:22:37.790880 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.790859 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.791500 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.791139 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m2jhf\"" Jul 29 06:22:37.791963 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.791937 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.792200 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.792176 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4wlg7\"" Jul 29 06:22:37.792709 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.792524 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.792709 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.792555 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:37.792709 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:37.792625 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793199 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1efa9f84-b540-41ee-b657-cbe318eff42c-host\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793232 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-lib-modules\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-etc-kubernetes\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793282 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-socket-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793307 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1efa9f84-b540-41ee-b657-cbe318eff42c-serviceca\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-kubelet\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793354 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8qh\" (UniqueName: \"kubernetes.io/projected/8913b5e8-911f-460d-8d41-173b88e17090-kube-api-access-xc8qh\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.793381 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793382 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8dm\" (UniqueName: \"kubernetes.io/projected/1554b36a-b841-4fb3-a5f5-de77b6fd508a-kube-api-access-7c8dm\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793412 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-sys-fs\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-host\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-cni-binary-copy\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-konnectivity-ca\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793591 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793616 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgg2f\" (UniqueName: \"kubernetes.io/projected/ac659c38-b483-4375-9bb0-4f05e89caebf-kube-api-access-fgg2f\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hnf\" (UniqueName: \"kubernetes.io/projected/7855df25-c8b9-42bc-8f89-3b409d64d952-kube-api-access-m2hnf\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793681 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-multus-certs\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1554b36a-b841-4fb3-a5f5-de77b6fd508a-iptables-alerter-script\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793742 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793765 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793814 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-tuned\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.793856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793838 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-modprobe-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-os-release\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-conf-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793925 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-systemd\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-k8s-cni-cncf-io\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.793973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrfx\" (UniqueName: \"kubernetes.io/projected/1efa9f84-b540-41ee-b657-cbe318eff42c-kube-api-access-rdrfx\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794000 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-netns\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-kubernetes\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794050 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-conf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1554b36a-b841-4fb3-a5f5-de77b6fd508a-host-slash\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-device-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794139 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-cnibin\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wqq\" (UniqueName: \"kubernetes.io/projected/cfad06fb-db01-4906-baf6-92c769990810-kube-api-access-c5wqq\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-sys\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794344 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794371 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-system-cni-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysconfig\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.794606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-run\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-multus\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-registration-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-os-release\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfad06fb-db01-4906-baf6-92c769990810-hosts-file\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794615 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-hostroot\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-agent-certs\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794687 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-cnibin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-socket-dir-parent\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-tmp\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dsf\" (UniqueName: \"kubernetes.io/projected/9256ba99-f0f3-42ca-973c-1000c7c3cec2-kube-api-access-27dsf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794778 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-system-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794802 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:37.794860 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-bin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.795667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794936 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad06fb-db01-4906-baf6-92c769990810-tmp-dir\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-var-lib-kubelet\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.794984 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-multus-daemon-config\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.795209 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:37.795263 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.795831 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.796163 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cwk2\"" Jul 29 06:22:37.796519 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.796365 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 06:22:37.796894 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.796599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 06:22:37.796894 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.796863 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 06:22:37.798624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.798597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 06:22:37.798999 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.798843 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 06:22:37.804530 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.804323 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 06:22:37.822176 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.821313 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2844m" Jul 29 06:22:37.829031 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.828738 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2844m" Jul 29 06:22:37.829031 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.828829 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:37.883527 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.883500 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895178 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hnf\" (UniqueName: \"kubernetes.io/projected/7855df25-c8b9-42bc-8f89-3b409d64d952-kube-api-access-m2hnf\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-multus-certs\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1554b36a-b841-4fb3-a5f5-de77b6fd508a-iptables-alerter-script\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-netns\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895340 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8z6\" (UniqueName: \"kubernetes.io/projected/312263e9-3826-4b13-8579-931bf52a49cf-kube-api-access-7c8z6\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895359 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-multus-certs\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895366 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.895430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895438 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-tuned\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-var-lib-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895514 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-etc-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-node-log\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-modprobe-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-os-release\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-conf-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895605 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-kubelet-config\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-systemd\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-k8s-cni-cncf-io\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-ovn\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895709 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrfx\" (UniqueName: \"kubernetes.io/projected/1efa9f84-b540-41ee-b657-cbe318eff42c-kube-api-access-rdrfx\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-netns\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895742 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-kubelet\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895759 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-log-socket\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.896150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895773 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-env-overrides\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895789 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-kubernetes\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895806 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-conf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1554b36a-b841-4fb3-a5f5-de77b6fd508a-host-slash\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895838 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-device-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895858 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895874 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-bin\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-dbus\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-cnibin\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wqq\" (UniqueName: \"kubernetes.io/projected/cfad06fb-db01-4906-baf6-92c769990810-kube-api-access-c5wqq\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895938 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895966 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895986 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-sys\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.895985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896017 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-system-cni-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysconfig\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897032 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896088 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-run\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896198 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-conf-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896225 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896239 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-netns\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896264 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-cnibin\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-kubernetes\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896524 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-sys\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896611 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysctl-conf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1554b36a-b841-4fb3-a5f5-de77b6fd508a-iptables-alerter-script\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896732 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-system-cni-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-sysconfig\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896793 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-os-release\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896841 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-systemd\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896847 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1554b36a-b841-4fb3-a5f5-de77b6fd508a-host-slash\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896862 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-modprobe-d\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-run-k8s-cni-cncf-io\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.897706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896915 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-run\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-device-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.896991 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-multus\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-registration-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897072 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-multus\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897120 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-os-release\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897153 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfad06fb-db01-4906-baf6-92c769990810-hosts-file\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-os-release\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897182 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-systemd-units\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-config\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-hostroot\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-agent-certs\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-cnibin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-socket-dir-parent\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-tmp\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dsf\" (UniqueName: \"kubernetes.io/projected/9256ba99-f0f3-42ca-973c-1000c7c3cec2-kube-api-access-27dsf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.898537 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-system-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-hostroot\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897123 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-registration-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897493 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cfad06fb-db01-4906-baf6-92c769990810-hosts-file\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-multus-socket-dir-parent\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897627 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7855df25-c8b9-42bc-8f89-3b409d64d952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897857 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-bin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897902 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad06fb-db01-4906-baf6-92c769990810-tmp-dir\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-systemd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897936 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-script-lib\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-var-lib-kubelet\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-multus-daemon-config\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-system-cni-dir\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.897986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-netd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1efa9f84-b540-41ee-b657-cbe318eff42c-host\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.899370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-lib-modules\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-etc-kubernetes\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-socket-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1efa9f84-b540-41ee-b657-cbe318eff42c-serviceca\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-kubelet\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8qh\" (UniqueName: \"kubernetes.io/projected/8913b5e8-911f-460d-8d41-173b88e17090-kube-api-access-xc8qh\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898147 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8dm\" (UniqueName: \"kubernetes.io/projected/1554b36a-b841-4fb3-a5f5-de77b6fd508a-kube-api-access-7c8dm\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898164 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-sys-fs\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898180 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mjk\" (UniqueName: \"kubernetes.io/projected/e17be3de-940a-4138-8d18-94f587b8e655-kube-api-access-r6mjk\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898198 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/312263e9-3826-4b13-8579-931bf52a49cf-ovn-node-metrics-cert\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898214 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-host\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898230 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-cni-binary-copy\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898265 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-konnectivity-ca\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgg2f\" (UniqueName: \"kubernetes.io/projected/ac659c38-b483-4375-9bb0-4f05e89caebf-kube-api-access-fgg2f\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898315 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-slash\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.912125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898338 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898446 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-cni-bin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-cnibin\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898862 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad06fb-db01-4906-baf6-92c769990810-tmp-dir\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.898865 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-var-lib-kubelet\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899038 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7855df25-c8b9-42bc-8f89-3b409d64d952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-multus-daemon-config\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1efa9f84-b540-41ee-b657-cbe318eff42c-host\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899436 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8913b5e8-911f-460d-8d41-173b88e17090-cni-binary-copy\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-lib-modules\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-etc-kubernetes\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-socket-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899638 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-konnectivity-ca\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899720 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9256ba99-f0f3-42ca-973c-1000c7c3cec2-host\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899774 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac659c38-b483-4375-9bb0-4f05e89caebf-sys-fs\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1efa9f84-b540-41ee-b657-cbe318eff42c-serviceca\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.913023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.899956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8913b5e8-911f-460d-8d41-173b88e17090-host-var-lib-kubelet\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.900337 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-etc-tuned\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.902577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9256ba99-f0f3-42ca-973c-1000c7c3cec2-tmp\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.907046 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hnf\" (UniqueName: \"kubernetes.io/projected/7855df25-c8b9-42bc-8f89-3b409d64d952-kube-api-access-m2hnf\") pod \"multus-additional-cni-plugins-9r4bn\" (UID: \"7855df25-c8b9-42bc-8f89-3b409d64d952\") " pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.907051 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrfx\" (UniqueName: \"kubernetes.io/projected/1efa9f84-b540-41ee-b657-cbe318eff42c-kube-api-access-rdrfx\") pod \"node-ca-6hn27\" (UID: \"1efa9f84-b540-41ee-b657-cbe318eff42c\") " pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.907393 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wqq\" (UniqueName: \"kubernetes.io/projected/cfad06fb-db01-4906-baf6-92c769990810-kube-api-access-c5wqq\") pod \"node-resolver-tg47d\" (UID: \"cfad06fb-db01-4906-baf6-92c769990810\") " pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.908112 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1-agent-certs\") pod \"konnectivity-agent-c2ghx\" (UID: \"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1\") " pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.910250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8qh\" (UniqueName: \"kubernetes.io/projected/8913b5e8-911f-460d-8d41-173b88e17090-kube-api-access-xc8qh\") pod \"multus-789b6\" (UID: \"8913b5e8-911f-460d-8d41-173b88e17090\") " pod="openshift-multus/multus-789b6" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.910382 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dsf\" (UniqueName: \"kubernetes.io/projected/9256ba99-f0f3-42ca-973c-1000c7c3cec2-kube-api-access-27dsf\") pod \"tuned-46lfr\" (UID: \"9256ba99-f0f3-42ca-973c-1000c7c3cec2\") " pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.910606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8dm\" (UniqueName: \"kubernetes.io/projected/1554b36a-b841-4fb3-a5f5-de77b6fd508a-kube-api-access-7c8dm\") pod \"iptables-alerter-nw6mw\" (UID: \"1554b36a-b841-4fb3-a5f5-de77b6fd508a\") " pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:37.914213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.911372 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgg2f\" (UniqueName: \"kubernetes.io/projected/ac659c38-b483-4375-9bb0-4f05e89caebf-kube-api-access-fgg2f\") pod \"aws-ebs-csi-driver-node-jkxdw\" (UID: \"ac659c38-b483-4375-9bb0-4f05e89caebf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:37.927100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.926860 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" event={"ID":"664c079835b1aca0098a8fa9d9f98d07","Type":"ContainerStarted","Data":"6c8d760a9f9944884825c2da4fd4451d6534d2c9db051a46f4dbc232662bb180"} Jul 29 06:22:37.928896 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.928870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"9c99cada63a53cc0230aa733d5f30bd1","Type":"ContainerStarted","Data":"4c32467037b9432461841cd0d38ba868bf257086f16d92e3db47a8f65a082792"} Jul 29 06:22:37.999538 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999501 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-systemd-units\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999538 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-config\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-systemd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-script-lib\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-netd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999667 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mjk\" (UniqueName: \"kubernetes.io/projected/e17be3de-940a-4138-8d18-94f587b8e655-kube-api-access-r6mjk\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/312263e9-3826-4b13-8579-931bf52a49cf-ovn-node-metrics-cert\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999717 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-slash\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999743 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:37.999784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999782 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-netns\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8z6\" (UniqueName: \"kubernetes.io/projected/312263e9-3826-4b13-8579-931bf52a49cf-kube-api-access-7c8z6\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-var-lib-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999848 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-etc-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-node-log\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999893 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-kubelet-config\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-ovn\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999972 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-kubelet\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:37.999995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-log-socket\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-env-overrides\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000047 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-bin\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000079 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-dbus\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000096 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.000218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000197 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-var-lib-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-etc-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-systemd-units\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000324 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-node-log\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000372 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-kubelet-config\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.000488 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.000578 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:38.500539291 +0000 UTC m=+3.034618872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-config\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.000997 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-systemd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001113 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-env-overrides\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001405 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b21ba907-5748-4c3f-b4d6-d8ee66073526-dbus\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-bin\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-ovn\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001536 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/312263e9-3826-4b13-8579-931bf52a49cf-ovnkube-script-lib\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001553 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-kubelet\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.001568 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-cni-netd\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-run-openvswitch\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.001618 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:38.501601181 +0000 UTC m=+3.035680734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001638 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001681 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-run-netns\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.001866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-host-slash\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.002192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.001968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/312263e9-3826-4b13-8579-931bf52a49cf-log-socket\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.004447 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.004423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/312263e9-3826-4b13-8579-931bf52a49cf-ovn-node-metrics-cert\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.008136 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.008018 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:38.008136 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.008040 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:38.008136 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.008055 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:38.008136 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.008111 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:38.508093651 +0000 UTC m=+3.042173225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:38.011155 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.011128 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mjk\" (UniqueName: \"kubernetes.io/projected/e17be3de-940a-4138-8d18-94f587b8e655-kube-api-access-r6mjk\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:38.011755 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.011733 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8z6\" (UniqueName: \"kubernetes.io/projected/312263e9-3826-4b13-8579-931bf52a49cf-kube-api-access-7c8z6\") pod \"ovnkube-node-ksftp\" (UID: \"312263e9-3826-4b13-8579-931bf52a49cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.104102 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.104030 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" Jul 29 06:22:38.107326 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.107303 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hn27" Jul 29 06:22:38.117538 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.117509 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46lfr" Jul 29 06:22:38.125517 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.125496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-789b6" Jul 29 06:22:38.134307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.134286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" Jul 29 06:22:38.141319 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.141301 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:22:38.148238 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.148213 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nw6mw" Jul 29 06:22:38.157108 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.157086 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg47d" Jul 29 06:22:38.164792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.164769 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:22:38.418974 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.418949 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 06:22:38.504648 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.504617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:38.504648 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.504659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:38.504900 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.504804 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:38.504900 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.504824 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:38.504900 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.504889 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.504868862 +0000 UTC m=+4.038948418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:38.505034 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.504906 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.504899119 +0000 UTC m=+4.038978671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:38.605544 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.605510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:38.605713 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.605688 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:38.605713 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.605709 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:38.605828 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.605720 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:38.605828 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.605781 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.605760258 +0000 UTC m=+4.139839833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:38.831276 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.831230 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 06:17:38 +0000 UTC" deadline="2028-01-22 15:15:43.754221786 +0000 UTC" Jul 29 06:22:38.831276 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.831270 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13016h53m4.922955977s" Jul 29 06:22:38.923211 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.922710 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:38.923211 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:38.922718 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:38.923211 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.922843 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:38.923211 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:38.922960 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:39.111867 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.111839 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1554b36a_b841_4fb3_a5f5_de77b6fd508a.slice/crio-26e0c6018f3b0f61e36e0fc9a79ff1ba8f4d4bdd6df94d627d3468986d2ba2e1 WatchSource:0}: Error finding container 26e0c6018f3b0f61e36e0fc9a79ff1ba8f4d4bdd6df94d627d3468986d2ba2e1: Status 404 returned error can't find the container with id 26e0c6018f3b0f61e36e0fc9a79ff1ba8f4d4bdd6df94d627d3468986d2ba2e1 Jul 29 06:22:39.128155 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.128133 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8913b5e8_911f_460d_8d41_173b88e17090.slice/crio-b8ec96b6354e200d171656ef81737d9a0617a7bcaab04e84018b8b7eb7e51a2c WatchSource:0}: Error finding container b8ec96b6354e200d171656ef81737d9a0617a7bcaab04e84018b8b7eb7e51a2c: Status 404 returned error can't find the container with id b8ec96b6354e200d171656ef81737d9a0617a7bcaab04e84018b8b7eb7e51a2c Jul 29 06:22:39.130834 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.130809 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9256ba99_f0f3_42ca_973c_1000c7c3cec2.slice/crio-df55c1709671187cc10c96562bd2b67ffe6d692e9c9a9f108029bb5c88fe4294 WatchSource:0}: Error finding container df55c1709671187cc10c96562bd2b67ffe6d692e9c9a9f108029bb5c88fe4294: Status 404 returned error can't find the container with id df55c1709671187cc10c96562bd2b67ffe6d692e9c9a9f108029bb5c88fe4294 Jul 29 06:22:39.131906 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.131886 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfad06fb_db01_4906_baf6_92c769990810.slice/crio-475e2cb36ceddb31cdc98fc5b5d26e35b8e6bd63e6c451ea6903e5256b2c52c2 WatchSource:0}: Error finding container 475e2cb36ceddb31cdc98fc5b5d26e35b8e6bd63e6c451ea6903e5256b2c52c2: Status 404 returned error can't find the container with id 475e2cb36ceddb31cdc98fc5b5d26e35b8e6bd63e6c451ea6903e5256b2c52c2 Jul 29 06:22:39.132967 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.132944 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd37587_97c4_4ee5_9a0c_db3d92b3f4a1.slice/crio-bd746f7c88740739f6baa2b64d6018e44859ce5fcb01f547daaa0509eca5502b WatchSource:0}: Error finding container bd746f7c88740739f6baa2b64d6018e44859ce5fcb01f547daaa0509eca5502b: Status 404 returned error can't find the container with id bd746f7c88740739f6baa2b64d6018e44859ce5fcb01f547daaa0509eca5502b Jul 29 06:22:39.134802 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.134779 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac659c38_b483_4375_9bb0_4f05e89caebf.slice/crio-3fbc389be3fb11f14295c9508ae842cc07d0963134095e8cbd18b79e3f462c32 WatchSource:0}: Error finding container 3fbc389be3fb11f14295c9508ae842cc07d0963134095e8cbd18b79e3f462c32: Status 404 returned error can't find the container with id 3fbc389be3fb11f14295c9508ae842cc07d0963134095e8cbd18b79e3f462c32 Jul 29 06:22:39.136661 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.135271 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7855df25_c8b9_42bc_8f89_3b409d64d952.slice/crio-52b9e1f333d8289a6bfecb16d701cf8c2196e7465d545a3923daca1c33ed1c8e WatchSource:0}: Error finding container 52b9e1f333d8289a6bfecb16d701cf8c2196e7465d545a3923daca1c33ed1c8e: Status 404 returned error can't find the container with id 52b9e1f333d8289a6bfecb16d701cf8c2196e7465d545a3923daca1c33ed1c8e Jul 29 06:22:39.137141 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.137119 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efa9f84_b540_41ee_b657_cbe318eff42c.slice/crio-99c93eeae8e0c0bb90dfff4b4b8f67a928a2fa16bb7779f1cff7131a57c428d6 WatchSource:0}: Error finding container 99c93eeae8e0c0bb90dfff4b4b8f67a928a2fa16bb7779f1cff7131a57c428d6: Status 404 returned error can't find the container with id 99c93eeae8e0c0bb90dfff4b4b8f67a928a2fa16bb7779f1cff7131a57c428d6 Jul 29 06:22:39.138849 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:22:39.138787 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312263e9_3826_4b13_8579_931bf52a49cf.slice/crio-a8012443faf22e9693a019bbe2763986e6431dbbc09b318a9ff1893a461bae62 WatchSource:0}: Error finding container a8012443faf22e9693a019bbe2763986e6431dbbc09b318a9ff1893a461bae62: Status 404 returned error can't find the container with id a8012443faf22e9693a019bbe2763986e6431dbbc09b318a9ff1893a461bae62 Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.511796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.512058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.511949 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.512287 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:41.512262208 +0000 UTC m=+6.046341761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.512223 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:39.512740 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.512647 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:41.512627587 +0000 UTC m=+6.046707158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:39.613569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.613532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:39.613759 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.613729 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:39.613759 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.613756 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:39.613878 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.613773 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:39.613878 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.613844 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:41.613824701 +0000 UTC m=+6.147904271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:39.832664 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.832541 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 06:17:38 +0000 UTC" deadline="2028-04-30 20:04:20.302580932 +0000 UTC" Jul 29 06:22:39.832664 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.832612 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15397h41m40.469973459s" Jul 29 06:22:39.924507 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.924431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:39.924685 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:39.924621 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:39.939808 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.939771 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hn27" event={"ID":"1efa9f84-b540-41ee-b657-cbe318eff42c","Type":"ContainerStarted","Data":"99c93eeae8e0c0bb90dfff4b4b8f67a928a2fa16bb7779f1cff7131a57c428d6"} Jul 29 06:22:39.945588 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.945534 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nw6mw" event={"ID":"1554b36a-b841-4fb3-a5f5-de77b6fd508a","Type":"ContainerStarted","Data":"26e0c6018f3b0f61e36e0fc9a79ff1ba8f4d4bdd6df94d627d3468986d2ba2e1"} Jul 29 06:22:39.955566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.955539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" event={"ID":"664c079835b1aca0098a8fa9d9f98d07","Type":"ContainerStarted","Data":"d26e85e2220205a05c780f63bfca97d3ad6d2bc8baca4e03c6c4f6be7a05c4cf"} Jul 29 06:22:39.964471 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.964431 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerStarted","Data":"52b9e1f333d8289a6bfecb16d701cf8c2196e7465d545a3923daca1c33ed1c8e"} Jul 29 06:22:39.969186 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.969154 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" event={"ID":"ac659c38-b483-4375-9bb0-4f05e89caebf","Type":"ContainerStarted","Data":"3fbc389be3fb11f14295c9508ae842cc07d0963134095e8cbd18b79e3f462c32"} Jul 29 06:22:39.973902 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.973873 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2ghx" event={"ID":"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1","Type":"ContainerStarted","Data":"bd746f7c88740739f6baa2b64d6018e44859ce5fcb01f547daaa0509eca5502b"} Jul 29 06:22:39.983862 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.983834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46lfr" event={"ID":"9256ba99-f0f3-42ca-973c-1000c7c3cec2","Type":"ContainerStarted","Data":"df55c1709671187cc10c96562bd2b67ffe6d692e9c9a9f108029bb5c88fe4294"} Jul 29 06:22:39.988691 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.988662 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg47d" event={"ID":"cfad06fb-db01-4906-baf6-92c769990810","Type":"ContainerStarted","Data":"475e2cb36ceddb31cdc98fc5b5d26e35b8e6bd63e6c451ea6903e5256b2c52c2"} Jul 29 06:22:39.997147 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:39.997125 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-789b6" event={"ID":"8913b5e8-911f-460d-8d41-173b88e17090","Type":"ContainerStarted","Data":"b8ec96b6354e200d171656ef81737d9a0617a7bcaab04e84018b8b7eb7e51a2c"} Jul 29 06:22:40.010739 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.010707 2556 generic.go:356] "Generic (PLEG): container finished" podID="9c99cada63a53cc0230aa733d5f30bd1" containerID="ceda2d240f4c45f6ee3a1c1fa8694d6061cdbffbcf5877c2fb1786f266c90f73" exitCode=0 Jul 29 06:22:40.011386 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.011347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"9c99cada63a53cc0230aa733d5f30bd1","Type":"ContainerDied","Data":"ceda2d240f4c45f6ee3a1c1fa8694d6061cdbffbcf5877c2fb1786f266c90f73"} Jul 29 06:22:40.028871 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.028819 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-31.ec2.internal" podStartSLOduration=3.02880231 podStartE2EDuration="3.02880231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:39.973389665 +0000 UTC m=+4.507469245" watchObservedRunningTime="2026-07-29 06:22:40.02880231 +0000 UTC m=+4.562881903" Jul 29 06:22:40.029577 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.029480 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"a8012443faf22e9693a019bbe2763986e6431dbbc09b318a9ff1893a461bae62"} Jul 29 06:22:40.922415 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.921638 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:40.922415 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:40.921771 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:40.922415 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:40.922235 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:40.922415 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:40.922356 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:41.048386 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.048348 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" event={"ID":"9c99cada63a53cc0230aa733d5f30bd1","Type":"ContainerStarted","Data":"7783e935b1be6cbe6984553852f090c8a37a0a676893361bd205710df816415e"} Jul 29 06:22:41.064743 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.064174 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-31.ec2.internal" podStartSLOduration=4.064151534 podStartE2EDuration="4.064151534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:41.063950212 +0000 UTC m=+5.598029790" watchObservedRunningTime="2026-07-29 06:22:41.064151534 +0000 UTC m=+5.598231112" Jul 29 06:22:41.529860 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.529049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:41.529860 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.529104 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:41.529860 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.529320 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:41.529860 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.529394 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:45.529365675 +0000 UTC m=+10.063445243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:41.529860 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.529844 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:41.530282 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.529899 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:45.529882058 +0000 UTC m=+10.063961627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:41.630291 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.630247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:41.630495 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.630474 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:41.630579 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.630496 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:41.630579 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.630511 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:41.630579 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.630574 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:45.630553802 +0000 UTC m=+10.164633380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:41.924628 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:41.924538 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:41.925252 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:41.924682 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:42.922835 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:42.921886 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:42.922835 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:42.922097 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:42.922835 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:42.922649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:42.922835 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:42.922782 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:43.924731 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:43.924269 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:43.924731 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:43.924382 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:44.923166 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:44.922479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:44.923166 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:44.922606 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:44.923166 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:44.923009 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:44.923166 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:44.923109 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:45.579340 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:45.579266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:45.579340 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:45.579324 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:45.579819 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.579476 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:45.579819 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.579535 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:45.579819 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.579564 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:53.57954071 +0000 UTC m=+18.113620272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:45.579819 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.579588 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:53.579573408 +0000 UTC m=+18.113652964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:45.680746 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:45.680085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:45.680746 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.680248 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:45.680746 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.680271 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:45.680746 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.680286 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:45.680746 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.680351 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:53.680332355 +0000 UTC m=+18.214411911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:45.922594 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:45.922510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:45.922757 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:45.922611 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:46.922202 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:46.921554 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:46.922202 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:46.921676 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:46.922202 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:46.922050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:46.922202 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:46.922160 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:47.922248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:47.922216 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:47.922724 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:47.922358 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:48.922122 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:48.922084 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:48.922311 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:48.922129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:48.922311 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:48.922236 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:48.922724 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:48.922685 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:49.921998 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:49.921958 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:49.922135 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:49.922091 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:50.922828 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:50.922391 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:50.922828 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:50.922523 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:50.922828 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:50.922399 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:50.922828 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:50.922691 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:51.922222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:51.922182 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:51.922392 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:51.922323 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:52.922600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:52.922567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:52.922600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:52.922608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:52.923123 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:52.922685 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:52.923123 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:52.922830 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:53.640765 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:53.640726 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:53.640765 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:53.640761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:53.640981 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.640886 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:53.640981 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.640894 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:53.640981 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.640944 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:09.640924691 +0000 UTC m=+34.175004243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:22:53.640981 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.640964 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:09.640954306 +0000 UTC m=+34.175033857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:22:53.741549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:53.741518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:53.741726 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.741667 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:22:53.741726 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.741684 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:22:53.741726 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.741698 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:53.741887 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.741758 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:09.741741571 +0000 UTC m=+34.275821146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:22:53.921991 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:53.921898 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:53.922146 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:53.922031 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:54.921929 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:54.921835 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:54.922359 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:54.921858 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:54.922359 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:54.921959 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:54.922359 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:54.922025 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:55.922838 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:55.922800 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:55.923254 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:55.922925 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:56.921601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:56.921563 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:56.921601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:56.921586 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:56.921851 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:56.921695 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:56.921908 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:56.921855 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:57.922607 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:57.922570 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:57.923115 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:57.922721 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:22:58.924438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:58.923956 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:22:58.924438 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:58.924275 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:22:58.924438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:58.924312 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:22:58.925081 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:58.925029 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:22:59.089899 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.089724 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="d49ff82302a04494d1e573f21384463513a886704f55de6c77450c0be3a79192" exitCode=0 Jul 29 06:22:59.090065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.089805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"d49ff82302a04494d1e573f21384463513a886704f55de6c77450c0be3a79192"} Jul 29 06:22:59.091581 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.091556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" event={"ID":"ac659c38-b483-4375-9bb0-4f05e89caebf","Type":"ContainerStarted","Data":"30b637fa15536c08713b33b65c63f851fb7fce3a293ac3cbcb655d83604e4e0e"} Jul 29 06:22:59.092953 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.092932 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2ghx" event={"ID":"5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1","Type":"ContainerStarted","Data":"4cce26467cfc6bee4e471ef00f44238b95a033790787d3862a5e39f338c67955"} Jul 29 06:22:59.094309 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.094287 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46lfr" event={"ID":"9256ba99-f0f3-42ca-973c-1000c7c3cec2","Type":"ContainerStarted","Data":"5d2b59b12e435b3e3a3c058040a863d0ee52ad639aa8be4550257647b25cd854"} Jul 29 06:22:59.095736 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.095695 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg47d" event={"ID":"cfad06fb-db01-4906-baf6-92c769990810","Type":"ContainerStarted","Data":"b5141e1a571b9c7ffcb6d37bb9d8a9e97ae06e0e60bd7f9862e392765d4aab2e"} Jul 29 06:22:59.097174 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.097156 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-789b6" event={"ID":"8913b5e8-911f-460d-8d41-173b88e17090","Type":"ContainerStarted","Data":"8545e9d5dc225b30a5ec36d8016fce1e89c78fa551c4603b3bd23396059d1e5b"} Jul 29 06:22:59.099119 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.099097 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"ca708002b69cf386af46ead9d4fd0afeadc12b9c78b141bfe3fab168b8da72a5"} Jul 29 06:22:59.099197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.099154 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"ea1e99038bf5d4bf649f4d05c6b47094d9a137527a36d76213827561459dd025"} Jul 29 06:22:59.100292 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.100275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hn27" event={"ID":"1efa9f84-b540-41ee-b657-cbe318eff42c","Type":"ContainerStarted","Data":"dd4a0ac1964100f50b42925c793af147146623f2496b9d0796082dd0239fd41b"} Jul 29 06:22:59.122638 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.122603 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6hn27" podStartSLOduration=3.558202119 podStartE2EDuration="23.122591812s" totalImagesPullingTime="19.564389693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.122588967 +0000 UTC m=+23.656668541" watchObservedRunningTime="2026-07-29 06:22:59.122591812 +0000 UTC m=+23.656671384" Jul 29 06:22:59.137278 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.137238 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-46lfr" podStartSLOduration=3.5791426729999998 podStartE2EDuration="23.137224748s" totalImagesPullingTime="19.558082075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.137143157 +0000 UTC m=+23.671222729" watchObservedRunningTime="2026-07-29 06:22:59.137224748 +0000 UTC m=+23.671304320" Jul 29 06:22:59.155699 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.155661 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-789b6" podStartSLOduration=3.536530193 podStartE2EDuration="23.155649104s" totalImagesPullingTime="19.619118911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.155077866 +0000 UTC m=+23.689157438" watchObservedRunningTime="2026-07-29 06:22:59.155649104 +0000 UTC m=+23.689728677" Jul 29 06:22:59.176574 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.176527 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c2ghx" podStartSLOduration=3.625344619 podStartE2EDuration="23.176507539s" totalImagesPullingTime="19.55116292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.175264457 +0000 UTC m=+23.709344032" watchObservedRunningTime="2026-07-29 06:22:59.176507539 +0000 UTC m=+23.710587180" Jul 29 06:22:59.192838 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.192802 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tg47d" podStartSLOduration=3.639692097 podStartE2EDuration="23.192789709s" totalImagesPullingTime="19.553097612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.192158265 +0000 UTC m=+23.726237832" watchObservedRunningTime="2026-07-29 06:22:59.192789709 +0000 UTC m=+23.726869281" Jul 29 06:22:59.922425 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:22:59.922360 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:22:59.922723 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:22:59.922528 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:00.106332 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.106061 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"107be844a648a1dd5ac3f3514098b2064449a12688f3f3558ea08ef3c5b94b13"} Jul 29 06:23:00.106332 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.106164 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"cef6ab736731b9a4000addf9031e1170d778fa7f33b5d6b797c18b59497bd707"} Jul 29 06:23:00.106332 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.106219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"a9f4fc17c1dffdd2915806ffa966bc2c8a311a75187b635760059cee91a96de7"} Jul 29 06:23:00.106332 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.106266 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"a48b1f44105dbdeefdf83095fda3d593ff75bbb50ca042e1744f5b8701f86180"} Jul 29 06:23:00.361180 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.361111 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 06:23:00.857973 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.857783 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T06:23:00.361130265Z","UUID":"867e5e65-f885-405f-90e2-65cfdd411d28","Handler":null,"Name":"","Endpoint":""} Jul 29 06:23:00.859788 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.859756 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 06:23:00.859788 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.859793 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 06:23:00.922437 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.922405 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:00.922598 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:00.922530 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:00.922868 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:00.922842 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:00.922979 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:00.922957 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:01.110873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:01.110439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nw6mw" event={"ID":"1554b36a-b841-4fb3-a5f5-de77b6fd508a","Type":"ContainerStarted","Data":"446669f41210fce41974a33de233f473a8205f590ed6ae5850e00710e2ba3b1f"} Jul 29 06:23:01.113143 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:01.113114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" event={"ID":"ac659c38-b483-4375-9bb0-4f05e89caebf","Type":"ContainerStarted","Data":"2348ede43d6546d54a29bbbdbc4476f565f36677e3fb690cf32cd1f20029a8e4"} Jul 29 06:23:01.124343 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:01.124292 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nw6mw" podStartSLOduration=5.541199036 podStartE2EDuration="25.124277388s" totalImagesPullingTime="19.583078352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:01.12416122 +0000 UTC m=+25.658240794" watchObservedRunningTime="2026-07-29 06:23:01.124277388 +0000 UTC m=+25.658356961" Jul 29 06:23:01.922370 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:01.922292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:01.922554 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:01.922444 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:02.120093 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:02.120050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"4bd06b1114abebfd7b182b2e629519c771f13560b0e86a1754e769af5ac120b7"} Jul 29 06:23:02.122269 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:02.122240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" event={"ID":"ac659c38-b483-4375-9bb0-4f05e89caebf","Type":"ContainerStarted","Data":"5e833ff0405f808d70458977c20eec348afad0ce2151cd279d089a4a290e21e0"} Jul 29 06:23:02.921600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:02.921573 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:02.921600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:02.921599 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:02.921804 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:02.921700 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:02.921869 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:02.921845 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:03.223625 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:03.223555 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:23:03.224239 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:03.224216 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:23:03.238052 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:03.238007 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkxdw" podStartSLOduration=5.38123575 podStartE2EDuration="27.237989677s" totalImagesPullingTime="21.856753927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:02.143404444 +0000 UTC m=+26.677484033" watchObservedRunningTime="2026-07-29 06:23:03.237989677 +0000 UTC m=+27.772069251" Jul 29 06:23:03.922641 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:03.922404 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:03.922896 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:03.922754 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:04.128844 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.128772 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="a7ae250a46d8a24ca23aab9f340f8751a0cabbb4ee53cce9a0f55d267a3bfacb" exitCode=0 Jul 29 06:23:04.129005 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.128866 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"a7ae250a46d8a24ca23aab9f340f8751a0cabbb4ee53cce9a0f55d267a3bfacb"} Jul 29 06:23:04.132598 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.132556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" event={"ID":"312263e9-3826-4b13-8579-931bf52a49cf","Type":"ContainerStarted","Data":"7fb25dd06623889ab26f319cba5db8dc4947b3cff3c9f27e0dccee7338324bb7"} Jul 29 06:23:04.133024 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.132831 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:23:04.133024 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.132870 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:04.133024 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.132924 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:04.133024 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.132972 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:04.133253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.133237 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c2ghx" Jul 29 06:23:04.148878 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.148854 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:04.149010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.148986 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:04.195700 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.195657 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" podStartSLOduration=7.562730326 podStartE2EDuration="27.195642643s" totalImagesPullingTime="19.632912317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:04.194146064 +0000 UTC m=+28.728225636" watchObservedRunningTime="2026-07-29 06:23:04.195642643 +0000 UTC m=+28.729722215" Jul 29 06:23:04.921744 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.921562 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:04.922584 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:04.921594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:04.922584 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:04.921843 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:04.922584 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:04.921877 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:05.824302 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.824234 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dg9h9"] Jul 29 06:23:05.824483 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.824345 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:05.824483 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:05.824458 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:05.827402 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.827360 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqvg9"] Jul 29 06:23:05.827541 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.827489 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:05.827627 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:05.827600 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:05.828164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.828039 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vfcd"] Jul 29 06:23:05.828164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:05.828124 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:05.828281 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:05.828212 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:06.139209 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:06.139125 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="852fdaad76e720d771a859e6e9ade1f3886054b03ded13487a4802998864a86c" exitCode=0 Jul 29 06:23:06.139209 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:06.139157 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"852fdaad76e720d771a859e6e9ade1f3886054b03ded13487a4802998864a86c"} Jul 29 06:23:07.925592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:07.925569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:07.925921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:07.925663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:07.925921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:07.925672 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:07.925921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:07.925690 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:07.925921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:07.925760 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:07.925921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:07.925824 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:08.146549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:08.146462 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="b224218438cec831714609a38198a189b5f183cfc3ee3dfb96fd3f53c88f7725" exitCode=0 Jul 29 06:23:08.146549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:08.146539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"b224218438cec831714609a38198a189b5f183cfc3ee3dfb96fd3f53c88f7725"} Jul 29 06:23:09.663941 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.663675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:09.664358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.663961 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:09.664358 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.663867 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 06:23:09.664358 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.664076 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret podName:b21ba907-5748-4c3f-b4d6-d8ee66073526 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:41.664059842 +0000 UTC m=+66.198139405 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret") pod "global-pull-secret-syncer-dg9h9" (UID: "b21ba907-5748-4c3f-b4d6-d8ee66073526") : object "kube-system"/"original-pull-secret" not registered Jul 29 06:23:09.664358 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.664093 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:23:09.664358 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.664157 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:41.66413956 +0000 UTC m=+66.198219137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 06:23:09.764606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.764572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:09.764763 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.764713 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 06:23:09.764763 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.764728 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 06:23:09.764763 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.764737 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mjmhm for pod openshift-network-diagnostics/network-check-target-5vfcd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:23:09.764958 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.764795 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm podName:a755fc7e-0b63-47dc-8a27-0baee6132a07 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:41.764778919 +0000 UTC m=+66.298858470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjmhm" (UniqueName: "kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm") pod "network-check-target-5vfcd" (UID: "a755fc7e-0b63-47dc-8a27-0baee6132a07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 06:23:09.922343 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.922262 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:09.922343 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.922278 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:09.922580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:09.922273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:09.922580 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.922412 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:09.922580 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.922474 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:09.922580 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:09.922560 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:11.922310 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:11.922238 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:11.922916 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:11.922246 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:11.922916 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:11.922395 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:11.922916 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:11.922401 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dg9h9" podUID="b21ba907-5748-4c3f-b4d6-d8ee66073526" Jul 29 06:23:11.922916 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:11.922510 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:23:11.922916 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:11.922616 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vfcd" podUID="a755fc7e-0b63-47dc-8a27-0baee6132a07" Jul 29 06:23:12.334977 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.334953 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-31.ec2.internal" event="NodeReady" Jul 29 06:23:12.335141 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.335129 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 06:23:12.378056 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.378026 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w9ltd"] Jul 29 06:23:12.379992 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.379966 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.384403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.381195 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xs5xp"] Jul 29 06:23:12.384403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.383363 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z5482\"" Jul 29 06:23:12.384403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.383875 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 06:23:12.384403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.384191 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 06:23:12.385774 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.385749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.388481 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.388463 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmx92\"" Jul 29 06:23:12.388666 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.388496 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 06:23:12.388743 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.388509 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 06:23:12.389035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.389011 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 06:23:12.390132 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.390112 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9ltd"] Jul 29 06:23:12.392869 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.392818 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs5xp"] Jul 29 06:23:12.485424 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485394 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.485546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485429 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df1d2617-2492-4ea0-b258-c2a27e18d731-cert\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.485546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485478 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-config-volume\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.485546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxg2\" (UniqueName: \"kubernetes.io/projected/df1d2617-2492-4ea0-b258-c2a27e18d731-kube-api-access-tpxg2\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.485546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485544 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-tmp-dir\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.485695 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.485563 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-kube-api-access-nzw8d\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.586784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586745 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df1d2617-2492-4ea0-b258-c2a27e18d731-cert\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.586949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586839 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-config-volume\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.586949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxg2\" (UniqueName: \"kubernetes.io/projected/df1d2617-2492-4ea0-b258-c2a27e18d731-kube-api-access-tpxg2\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.586949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-tmp-dir\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.586949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-kube-api-access-nzw8d\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.587167 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.586963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.587167 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:12.587055 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:12.587167 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:12.587130 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:13.087109316 +0000 UTC m=+37.621188884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:12.587322 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.587298 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-tmp-dir\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.587624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.587607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-config-volume\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.591204 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.591185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df1d2617-2492-4ea0-b258-c2a27e18d731-cert\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.595428 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.595407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxg2\" (UniqueName: \"kubernetes.io/projected/df1d2617-2492-4ea0-b258-c2a27e18d731-kube-api-access-tpxg2\") pod \"ingress-canary-xs5xp\" (UID: \"df1d2617-2492-4ea0-b258-c2a27e18d731\") " pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.595545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.595529 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-kube-api-access-nzw8d\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:12.701933 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.701854 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs5xp" Jul 29 06:23:12.828443 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:12.828275 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs5xp"] Jul 29 06:23:12.832330 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:23:12.832307 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1d2617_2492_4ea0_b258_c2a27e18d731.slice/crio-857a3a12d725f7caa6b1c04af8148a492fa72d307bbad207f50b4d41e63b2783 WatchSource:0}: Error finding container 857a3a12d725f7caa6b1c04af8148a492fa72d307bbad207f50b4d41e63b2783: Status 404 returned error can't find the container with id 857a3a12d725f7caa6b1c04af8148a492fa72d307bbad207f50b4d41e63b2783 Jul 29 06:23:13.091133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.091016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:13.091822 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:13.091140 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:13.091822 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:13.091206 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:14.091188097 +0000 UTC m=+38.625267663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:13.158549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.158511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs5xp" event={"ID":"df1d2617-2492-4ea0-b258-c2a27e18d731","Type":"ContainerStarted","Data":"857a3a12d725f7caa6b1c04af8148a492fa72d307bbad207f50b4d41e63b2783"} Jul 29 06:23:13.160885 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.160859 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="3f2c284eac594067fab40501213191b03f97e34edb1b5af3d27c786ea5dc678a" exitCode=0 Jul 29 06:23:13.160990 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.160920 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"3f2c284eac594067fab40501213191b03f97e34edb1b5af3d27c786ea5dc678a"} Jul 29 06:23:13.922088 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.922053 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:13.922270 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.922051 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:13.922270 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.922249 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:13.924965 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.924943 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btmpr\"" Jul 29 06:23:13.925128 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.925105 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 06:23:13.926368 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.926346 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 06:23:13.926486 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.926418 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 06:23:13.926486 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.926434 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4kn4r\"" Jul 29 06:23:13.926589 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:13.926354 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 06:23:14.100633 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:14.100600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:14.101086 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:14.100755 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:14.101086 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:14.100824 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:16.100804139 +0000 UTC m=+40.634883713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:14.165594 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:14.165564 2556 generic.go:356] "Generic (PLEG): container finished" podID="7855df25-c8b9-42bc-8f89-3b409d64d952" containerID="7a608990f0e2b2069200a9d82364bc84c94863b2861eaf535f37681ee9b064d9" exitCode=0 Jul 29 06:23:14.165754 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:14.165624 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerDied","Data":"7a608990f0e2b2069200a9d82364bc84c94863b2861eaf535f37681ee9b064d9"} Jul 29 06:23:15.170549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.170512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" event={"ID":"7855df25-c8b9-42bc-8f89-3b409d64d952","Type":"ContainerStarted","Data":"da11bc4935844ca4e0179eb96d1559f3f66841dca3582e0c4e6ec9570dd204cd"} Jul 29 06:23:15.192789 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.192741 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9r4bn" podStartSLOduration=8.831338705 podStartE2EDuration="39.192725543s" totalImagesPullingTime="30.361386838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:15.19097209 +0000 UTC m=+39.725051660" watchObservedRunningTime="2026-07-29 06:23:15.192725543 +0000 UTC m=+39.726805115" Jul 29 06:23:15.224874 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.224817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:15.320794 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.320768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:15.371571 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.371544 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:15.425478 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.425300 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:15.817134 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.817053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:15.867807 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:15.867778 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:16.114239 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.114169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:16.114364 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:16.114309 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:16.114422 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:16.114369 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:20.114354067 +0000 UTC m=+44.648433620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:16.156036 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.156008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:16.177045 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.177010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs5xp" event={"ID":"df1d2617-2492-4ea0-b258-c2a27e18d731","Type":"ContainerStarted","Data":"e68bec10a17261fbb552ca23005726de4f3ab0f3532282b7ee88cf075c9d58a5"} Jul 29 06:23:16.193209 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.193168 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xs5xp" podStartSLOduration=1.643020968 podStartE2EDuration="4.193155713s" totalImagesPullingTime="2.550134745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:16.192201372 +0000 UTC m=+40.726280946" watchObservedRunningTime="2026-07-29 06:23:16.193155713 +0000 UTC m=+40.727235286" Jul 29 06:23:16.259035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.259004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:16.560616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.560538 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:16.662186 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:16.662159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:17.232023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:17.231993 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:18.260074 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:18.260038 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:18.330071 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:18.330042 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:20.139885 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:20.139848 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:20.140273 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:20.139993 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:20.140273 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:20.140051 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:28.140037974 +0000 UTC m=+52.674117528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:22.471471 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:22.471437 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:22.523428 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:22.523393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:28.195019 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:28.194978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:28.195499 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:28.195128 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:28.195499 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:28.195223 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:44.195207301 +0000 UTC m=+68.729286852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:36.149976 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:36.149944 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksftp" Jul 29 06:23:36.305735 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:36.305702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:36.354136 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:36.354107 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:39.208495 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:39.208456 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:39.259353 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:39.259326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:39.638511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:39.638425 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:39.687695 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:39.687672 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:39.737795 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:39.737776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:41.681307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.681265 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:41.681307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.681306 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:23:41.684808 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.684781 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 06:23:41.684890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.684797 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 06:23:41.692310 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:41.692285 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 06:23:41.692372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:41.692358 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:24:45.692336604 +0000 UTC m=+130.226416157 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : secret "metrics-daemon-secret" not found Jul 29 06:23:41.696045 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.696019 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b21ba907-5748-4c3f-b4d6-d8ee66073526-original-pull-secret\") pod \"global-pull-secret-syncer-dg9h9\" (UID: \"b21ba907-5748-4c3f-b4d6-d8ee66073526\") " pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:41.781809 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.781768 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:41.784546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.784531 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 06:23:41.795509 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.795484 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 06:23:41.806300 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.806277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmhm\" (UniqueName: \"kubernetes.io/projected/a755fc7e-0b63-47dc-8a27-0baee6132a07-kube-api-access-mjmhm\") pod \"network-check-target-5vfcd\" (UID: \"a755fc7e-0b63-47dc-8a27-0baee6132a07\") " pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:41.841302 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.841272 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dg9h9" Jul 29 06:23:41.850470 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.850444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4kn4r\"" Jul 29 06:23:41.858808 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.858788 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:41.966825 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.966761 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dg9h9"] Jul 29 06:23:41.969691 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:23:41.969661 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21ba907_5748_4c3f_b4d6_d8ee66073526.slice/crio-1ce11081842fc9fb965ef8eaa5db3bf9615aeb2bd1cdea6bf7c00fe69d0a4c7d WatchSource:0}: Error finding container 1ce11081842fc9fb965ef8eaa5db3bf9615aeb2bd1cdea6bf7c00fe69d0a4c7d: Status 404 returned error can't find the container with id 1ce11081842fc9fb965ef8eaa5db3bf9615aeb2bd1cdea6bf7c00fe69d0a4c7d Jul 29 06:23:41.983772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:41.983742 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vfcd"] Jul 29 06:23:41.985715 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:23:41.985688 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda755fc7e_0b63_47dc_8a27_0baee6132a07.slice/crio-b840d6adb58b8e0cbeacc3536b1502bc4b0a7a13b40b76aef589ea894767ec8d WatchSource:0}: Error finding container b840d6adb58b8e0cbeacc3536b1502bc4b0a7a13b40b76aef589ea894767ec8d: Status 404 returned error can't find the container with id b840d6adb58b8e0cbeacc3536b1502bc4b0a7a13b40b76aef589ea894767ec8d Jul 29 06:23:42.232241 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:42.232143 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vfcd" event={"ID":"a755fc7e-0b63-47dc-8a27-0baee6132a07","Type":"ContainerStarted","Data":"b840d6adb58b8e0cbeacc3536b1502bc4b0a7a13b40b76aef589ea894767ec8d"} Jul 29 06:23:42.233142 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:42.233120 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dg9h9" event={"ID":"b21ba907-5748-4c3f-b4d6-d8ee66073526","Type":"ContainerStarted","Data":"1ce11081842fc9fb965ef8eaa5db3bf9615aeb2bd1cdea6bf7c00fe69d0a4c7d"} Jul 29 06:23:44.201356 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:44.201312 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:23:44.201793 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:44.201521 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:23:44.201793 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:23:44.201605 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:24:16.201588027 +0000 UTC m=+100.735667577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:23:46.246730 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:46.246696 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vfcd" event={"ID":"a755fc7e-0b63-47dc-8a27-0baee6132a07","Type":"ContainerStarted","Data":"084b9ed9881247874c98b104f8f982be24628caf27f3a9c0710433abe4fa4501"} Jul 29 06:23:46.247220 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:46.246825 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:23:47.250592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:47.250491 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dg9h9" event={"ID":"b21ba907-5748-4c3f-b4d6-d8ee66073526","Type":"ContainerStarted","Data":"b7ca131d24dd56cefc2f59bade19e0daa441000597bbe2af78f295e42ab2d9cf"} Jul 29 06:23:47.265243 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:47.265199 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5vfcd" podStartSLOduration=66.774850268 podStartE2EDuration="1m10.265182375s" totalImagesPullingTime="3.490332107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:46.262522779 +0000 UTC m=+70.796602354" watchObservedRunningTime="2026-07-29 06:23:47.265182375 +0000 UTC m=+71.799261947" Jul 29 06:23:47.265412 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:47.265319 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dg9h9" podStartSLOduration=65.256892816 podStartE2EDuration="1m10.265315577s" totalImagesPullingTime="5.008422761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:47.264606097 +0000 UTC m=+71.798685670" watchObservedRunningTime="2026-07-29 06:23:47.265315577 +0000 UTC m=+71.799395149" Jul 29 06:23:51.918868 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:51.918829 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:23:51.969352 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:23:51.969327 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:24:07.085608 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:07.085476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:24:07.130140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:07.130105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:24:10.218578 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.218544 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt"] Jul 29 06:24:10.220593 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.220577 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.223428 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.223405 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 06:24:10.223621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.223436 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 06:24:10.224939 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.224915 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 06:24:10.225035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.224919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 06:24:10.230573 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.230554 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt"] Jul 29 06:24:10.268423 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.268397 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9"] Jul 29 06:24:10.270082 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.270068 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.272452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.272427 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f41b6ea-e61a-4986-a186-d8436005a73d-klusterlet-config\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.272548 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.272458 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f41b6ea-e61a-4986-a186-d8436005a73d-tmp\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.272548 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.272480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l47c\" (UniqueName: \"kubernetes.io/projected/5f41b6ea-e61a-4986-a186-d8436005a73d-kube-api-access-5l47c\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.272943 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.272926 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 06:24:10.273021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.272992 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 06:24:10.273021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.273002 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 06:24:10.273137 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.273123 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 06:24:10.281560 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.281541 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9"] Jul 29 06:24:10.373215 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373185 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d2becb-2b70-4e3c-a899-41ae81e2f5af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373215 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373217 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drp4\" (UniqueName: \"kubernetes.io/projected/95d2becb-2b70-4e3c-a899-41ae81e2f5af-kube-api-access-9drp4\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373241 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373262 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373407 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.373564 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f41b6ea-e61a-4986-a186-d8436005a73d-klusterlet-config\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.373564 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f41b6ea-e61a-4986-a186-d8436005a73d-tmp\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.373564 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l47c\" (UniqueName: \"kubernetes.io/projected/5f41b6ea-e61a-4986-a186-d8436005a73d-kube-api-access-5l47c\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.373879 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.373861 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f41b6ea-e61a-4986-a186-d8436005a73d-tmp\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.376167 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.376147 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f41b6ea-e61a-4986-a186-d8436005a73d-klusterlet-config\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.381478 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.381457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l47c\" (UniqueName: \"kubernetes.io/projected/5f41b6ea-e61a-4986-a186-d8436005a73d-kube-api-access-5l47c\") pod \"klusterlet-addon-workmgr-884f66c98-qbmvt\" (UID: \"5f41b6ea-e61a-4986-a186-d8436005a73d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.474189 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474189 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d2becb-2b70-4e3c-a899-41ae81e2f5af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drp4\" (UniqueName: \"kubernetes.io/projected/95d2becb-2b70-4e3c-a899-41ae81e2f5af-kube-api-access-9drp4\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.474994 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.474967 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d2becb-2b70-4e3c-a899-41ae81e2f5af-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.476649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.476622 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-ca\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.476755 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.476715 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.477232 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.477208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.477440 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.477418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/95d2becb-2b70-4e3c-a899-41ae81e2f5af-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.482182 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.482159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drp4\" (UniqueName: \"kubernetes.io/projected/95d2becb-2b70-4e3c-a899-41ae81e2f5af-kube-api-access-9drp4\") pod \"cluster-proxy-proxy-agent-84f77b8d5c-xwlt9\" (UID: \"95d2becb-2b70-4e3c-a899-41ae81e2f5af\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.529922 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.529901 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:10.585553 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.585521 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:24:10.655643 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.655607 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt"] Jul 29 06:24:10.658268 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:24:10.658241 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f41b6ea_e61a_4986_a186_d8436005a73d.slice/crio-fc217cfd76d05244c3c207b7ec86fe06d2d7060f32a01989012721b2a04bd711 WatchSource:0}: Error finding container fc217cfd76d05244c3c207b7ec86fe06d2d7060f32a01989012721b2a04bd711: Status 404 returned error can't find the container with id fc217cfd76d05244c3c207b7ec86fe06d2d7060f32a01989012721b2a04bd711 Jul 29 06:24:10.705549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:10.705522 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9"] Jul 29 06:24:10.707935 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:24:10.707913 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d2becb_2b70_4e3c_a899_41ae81e2f5af.slice/crio-4383856c163a04957928755eaaf2fbbbce80cf2e75dd065422df077a8b0e1cd4 WatchSource:0}: Error finding container 4383856c163a04957928755eaaf2fbbbce80cf2e75dd065422df077a8b0e1cd4: Status 404 returned error can't find the container with id 4383856c163a04957928755eaaf2fbbbce80cf2e75dd065422df077a8b0e1cd4 Jul 29 06:24:11.311106 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:11.311069 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerStarted","Data":"4383856c163a04957928755eaaf2fbbbce80cf2e75dd065422df077a8b0e1cd4"} Jul 29 06:24:11.312455 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:11.312421 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" event={"ID":"5f41b6ea-e61a-4986-a186-d8436005a73d","Type":"ContainerStarted","Data":"fc217cfd76d05244c3c207b7ec86fe06d2d7060f32a01989012721b2a04bd711"} Jul 29 06:24:16.219658 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.219618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:24:16.220054 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:24:16.219771 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:24:16.220054 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:24:16.219844 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:25:20.219829073 +0000 UTC m=+164.753908624 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:24:16.328632 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.328599 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" event={"ID":"5f41b6ea-e61a-4986-a186-d8436005a73d","Type":"ContainerStarted","Data":"d92bdc717f4748a60c3a1320e2746fca0e8e82f3e8c2e792f078b5fbaf4ddbb9"} Jul 29 06:24:16.328782 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.328688 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:16.330619 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.330597 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:24:16.330823 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.330799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerStarted","Data":"b0b1f00da888588ff01c9c79ab78b2ea72e741eac67590a6057b6d69ccd68043"} Jul 29 06:24:16.330908 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.330839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerStarted","Data":"692deaac999d9d63363fc221829e78bd9187d655acffd193077551d53a40da29"} Jul 29 06:24:16.330908 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.330853 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerStarted","Data":"e03af4b8295dcc81b7a859b70785a2e6127f1dd191bacd61b5b2b435e320ec29"} Jul 29 06:24:16.348120 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.348072 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" podStartSLOduration=1.206003791 podStartE2EDuration="6.348058573s" totalImagesPullingTime="5.142054782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:24:16.34708032 +0000 UTC m=+100.881159893" watchObservedRunningTime="2026-07-29 06:24:16.348058573 +0000 UTC m=+100.882138175" Jul 29 06:24:16.364028 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:16.363990 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" podStartSLOduration=1.269093646 podStartE2EDuration="6.363977464s" totalImagesPullingTime="5.094883818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:24:16.363798907 +0000 UTC m=+100.897878476" watchObservedRunningTime="2026-07-29 06:24:16.363977464 +0000 UTC m=+100.898057037" Jul 29 06:24:17.253222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:17.253192 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5vfcd" Jul 29 06:24:45.722314 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:24:45.722276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:24:45.722801 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:24:45.722408 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 06:24:45.722801 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:24:45.722468 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:47.722450583 +0000 UTC m=+252.256530139 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : secret "metrics-daemon-secret" not found Jul 29 06:25:15.396989 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:25:15.396950 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w9ltd" podUID="ce8b77c4-2fc8-4bd3-891d-94f96aca97d9" Jul 29 06:25:15.493101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:25:15.493071 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:25:16.934514 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:25:16.934475 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:25:20.245154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:25:20.245114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:25:20.245566 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:25:20.245263 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 06:25:20.245566 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:25:20.245340 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls podName:ce8b77c4-2fc8-4bd3-891d-94f96aca97d9 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:22.245325308 +0000 UTC m=+286.779404863 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls") pod "dns-default-w9ltd" (UID: "ce8b77c4-2fc8-4bd3-891d-94f96aca97d9") : secret "dns-default-metrics-tls" not found Jul 29 06:25:28.922251 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:25:28.922169 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:26:15.917277 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:15.917244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:16.330031 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:16.329986 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" podUID="5f41b6ea-e61a-4986-a186-d8436005a73d" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused" Jul 29 06:26:16.659988 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:16.659909 2556 generic.go:356] "Generic (PLEG): container finished" podID="5f41b6ea-e61a-4986-a186-d8436005a73d" containerID="d92bdc717f4748a60c3a1320e2746fca0e8e82f3e8c2e792f078b5fbaf4ddbb9" exitCode=1 Jul 29 06:26:16.659988 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:16.659973 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" event={"ID":"5f41b6ea-e61a-4986-a186-d8436005a73d","Type":"ContainerDied","Data":"d92bdc717f4748a60c3a1320e2746fca0e8e82f3e8c2e792f078b5fbaf4ddbb9"} Jul 29 06:26:16.660280 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:16.660264 2556 scope.go:117] "RemoveContainer" containerID="d92bdc717f4748a60c3a1320e2746fca0e8e82f3e8c2e792f078b5fbaf4ddbb9" Jul 29 06:26:17.664114 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:17.664081 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" event={"ID":"5f41b6ea-e61a-4986-a186-d8436005a73d","Type":"ContainerStarted","Data":"746ae7ce5816ecc521cbafd84f015c83e0a1c5d5f29d91df86f3352acaed6c95"} Jul 29 06:26:17.664518 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:17.664395 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:26:17.664981 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:17.664960 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-884f66c98-qbmvt" Jul 29 06:26:38.318897 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:38.318856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:39.707958 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:39.707914 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:40.687227 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:40.687189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:40.739874 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:40.739842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.172777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.172744 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-72skd"] Jul 29 06:26:42.174604 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.174587 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.177341 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.177316 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 06:26:42.177547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.177528 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75z2t\"" Jul 29 06:26:42.178858 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.178846 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 06:26:42.178906 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.178850 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 06:26:42.178906 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.178878 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 06:26:42.188715 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.188693 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72skd"] Jul 29 06:26:42.206796 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.206774 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.261291 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.261266 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.305423 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.305373 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc9deaae-8679-4bd7-905f-d3457077a28b-data-volume\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.305423 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.305422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.305624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.305443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.305624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.305499 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cbv\" (UniqueName: \"kubernetes.io/projected/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-api-access-82cbv\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.305624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.305587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc9deaae-8679-4bd7-905f-d3457077a28b-crio-socket\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.406949 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.406908 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cbv\" (UniqueName: \"kubernetes.io/projected/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-api-access-82cbv\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407089 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.406968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc9deaae-8679-4bd7-905f-d3457077a28b-crio-socket\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407089 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.407030 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bc9deaae-8679-4bd7-905f-d3457077a28b-crio-socket\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407199 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.407178 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc9deaae-8679-4bd7-905f-d3457077a28b-data-volume\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.407209 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.407230 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.407306 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.407293 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 06:26:42.407341 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.407307 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:42.407373 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.407360 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:42.907347591 +0000 UTC m=+247.441427158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : configmap "kube-rbac-proxy" not found Jul 29 06:26:42.407373 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.407372 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:42.907366597 +0000 UTC m=+247.441446148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:42.407524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.407500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc9deaae-8679-4bd7-905f-d3457077a28b-data-volume\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.417299 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.417267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cbv\" (UniqueName: \"kubernetes.io/projected/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-api-access-82cbv\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.674116 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.674089 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.770181 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.770149 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.871157 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.871122 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:42.912777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.912748 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.912870 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.912784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:42.912908 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.912881 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 06:26:42.912908 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.912888 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:42.912968 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.912943 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:43.912928584 +0000 UTC m=+248.447008135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:42.912968 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:42.912956 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:43.912949485 +0000 UTC m=+248.447029036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : configmap "kube-rbac-proxy" not found Jul 29 06:26:42.971896 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:42.971865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:43.390594 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:43.390514 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:43.491205 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:43.491172 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:43.920584 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:43.920538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:43.920584 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:43.920588 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:43.920921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:43.920658 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 06:26:43.920921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:43.920703 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:45.920690295 +0000 UTC m=+250.454769847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : configmap "kube-rbac-proxy" not found Jul 29 06:26:43.920921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:43.920704 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:43.920921 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:43.920766 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:45.920748575 +0000 UTC m=+250.454828140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:44.353794 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:44.353767 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg47d_cfad06fb-db01-4906-baf6-92c769990810/dns-node-resolver/0.log" Jul 29 06:26:44.953861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:44.953831 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hn27_1efa9f84-b540-41ee-b657-cbe318eff42c/node-ca/0.log" Jul 29 06:26:45.157959 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:45.157925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:45.256477 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:45.256347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:45.820362 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:45.820326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:45.937101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:45.937076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:45.937101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:45.937107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:45.937372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:45.937186 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 06:26:45.937372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:45.937217 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:45.937372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:45.937279 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:49.937262786 +0000 UTC m=+254.471342339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:45.937372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:45.937293 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:49.937286903 +0000 UTC m=+254.471366454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : configmap "kube-rbac-proxy" not found Jul 29 06:26:46.154692 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:46.154614 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs5xp_df1d2617-2492-4ea0-b258-c2a27e18d731/serve-healthcheck-canary/0.log" Jul 29 06:26:47.749246 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:47.749213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:26:47.749689 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:47.749348 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 06:26:47.749689 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:47.749422 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs podName:e17be3de-940a-4138-8d18-94f587b8e655 nodeName:}" failed. No retries permitted until 2026-07-29 06:28:49.749406748 +0000 UTC m=+374.283486301 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs") pod "network-metrics-daemon-dqvg9" (UID: "e17be3de-940a-4138-8d18-94f587b8e655") : secret "metrics-daemon-secret" not found Jul 29 06:26:48.305698 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:48.305665 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:48.904222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:48.904185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:49.023420 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.023366 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:49.125029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.124986 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:49.225246 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.225208 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:49.281439 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.281403 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:49.965630 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.965597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:49.966012 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:49.965679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:49.966012 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:49.965717 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 06:26:49.966012 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:49.965778 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:49.966012 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:49.965800 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:57.965784084 +0000 UTC m=+262.499863635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : configmap "kube-rbac-proxy" not found Jul 29 06:26:49.966012 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:49.965828 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:26:57.965817358 +0000 UTC m=+262.499896909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:50.003253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:50.003224 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:50.401337 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:50.401248 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:50.603472 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:50.603442 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:51.205460 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:51.205427 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:51.601670 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:51.601587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:52.056441 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:52.056405 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:52.400153 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:52.400071 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:52.660641 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:52.660557 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:52.802903 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:52.802871 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:52.904264 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:52.904230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:53.223776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:53.223739 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:53.603522 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:53.603438 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:53.802786 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:53.802754 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:54.400273 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:54.400236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.004850 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.004806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.198007 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.197962 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.381005 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.380913 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.481710 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.481676 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.597092 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.597050 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:55.802865 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:55.802836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:56.398369 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:56.398290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:56.797094 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:56.797063 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:57.003260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:57.003231 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:57.400597 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:57.400570 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:58.001617 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.001590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:58.024136 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.024114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:58.024205 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.024145 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:58.024272 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:58.024257 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 06:26:58.024328 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:26:58.024319 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls podName:bc9deaae-8679-4bd7-905f-d3457077a28b nodeName:}" failed. No retries permitted until 2026-07-29 06:27:14.024303189 +0000 UTC m=+278.558382743 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-72skd" (UID: "bc9deaae-8679-4bd7-905f-d3457077a28b") : secret "insights-runtime-extractor-tls" not found Jul 29 06:26:58.024688 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.024673 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bc9deaae-8679-4bd7-905f-d3457077a28b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:26:58.407183 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.407098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:58.597164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:58.597118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:59.001635 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:59.001597 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:59.402832 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:59.402757 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:26:59.601836 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:26:59.601801 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:02.820743 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:02.820711 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:06.278799 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.278765 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-566654d898-kc6wh"] Jul 29 06:27:06.281834 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.281815 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.286218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.286194 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 06:27:06.286891 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.286869 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 06:27:06.287019 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.286944 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 06:27:06.287747 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.287727 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtwx9\"" Jul 29 06:27:06.293151 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.293132 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 06:27:06.295301 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.295283 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566654d898-kc6wh"] Jul 29 06:27:06.338739 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.338719 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:06.439550 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.439520 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:06.480808 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtxj\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-kube-api-access-pmtxj\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.480911 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480812 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-tls\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.480911 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-trusted-ca\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.480995 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-installation-pull-secrets\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.480995 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480951 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-image-registry-private-configuration\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.480995 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.480980 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12733955-7a4c-4710-8cf2-f610aa9bcd8f-ca-trust-extracted\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.481101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.481019 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-bound-sa-token\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.481101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.481057 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-certificates\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-trusted-ca\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-installation-pull-secrets\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-image-registry-private-configuration\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12733955-7a4c-4710-8cf2-f610aa9bcd8f-ca-trust-extracted\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-bound-sa-token\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-certificates\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtxj\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-kube-api-access-pmtxj\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-tls\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.582920 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.582892 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12733955-7a4c-4710-8cf2-f610aa9bcd8f-ca-trust-extracted\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.583237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.583218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-trusted-ca\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.583357 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.583336 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-certificates\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.584886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.584866 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-image-registry-private-configuration\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.585295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.585272 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12733955-7a4c-4710-8cf2-f610aa9bcd8f-installation-pull-secrets\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.585408 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.585372 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-registry-tls\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.589947 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.589929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-bound-sa-token\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.590311 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.590292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtxj\" (UniqueName: \"kubernetes.io/projected/12733955-7a4c-4710-8cf2-f610aa9bcd8f-kube-api-access-pmtxj\") pod \"image-registry-566654d898-kc6wh\" (UID: \"12733955-7a4c-4710-8cf2-f610aa9bcd8f\") " pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.590466 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.590453 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.713993 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.713928 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566654d898-kc6wh"] Jul 29 06:27:06.716183 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:27:06.716143 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12733955_7a4c_4710_8cf2_f610aa9bcd8f.slice/crio-ac707b1e4e24537bf24d073e5c37bbd49a32cce5fe2082ea20cd996a97cd4cac WatchSource:0}: Error finding container ac707b1e4e24537bf24d073e5c37bbd49a32cce5fe2082ea20cd996a97cd4cac: Status 404 returned error can't find the container with id ac707b1e4e24537bf24d073e5c37bbd49a32cce5fe2082ea20cd996a97cd4cac Jul 29 06:27:06.799441 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.799410 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566654d898-kc6wh" event={"ID":"12733955-7a4c-4710-8cf2-f610aa9bcd8f","Type":"ContainerStarted","Data":"0a6dd862e52e36390631a7e75a06bc1baa15adf18bb8726970a230bac9766e8e"} Jul 29 06:27:06.799573 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.799456 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566654d898-kc6wh" event={"ID":"12733955-7a4c-4710-8cf2-f610aa9bcd8f","Type":"ContainerStarted","Data":"ac707b1e4e24537bf24d073e5c37bbd49a32cce5fe2082ea20cd996a97cd4cac"} Jul 29 06:27:06.799573 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.799505 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:06.818318 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:06.818276 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podStartSLOduration=0.818257606 podStartE2EDuration="818.257606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:06.817514236 +0000 UTC m=+271.351593809" watchObservedRunningTime="2026-07-29 06:27:06.818257606 +0000 UTC m=+271.352337182" Jul 29 06:27:07.476624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:07.476589 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:07.529824 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:07.529791 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:12.462894 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:12.462843 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:12.623248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:12.623212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:13.085187 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:13.085154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:14.033313 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:14.033277 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:27:14.035696 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:14.035680 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bc9deaae-8679-4bd7-905f-d3457077a28b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-72skd\" (UID: \"bc9deaae-8679-4bd7-905f-d3457077a28b\") " pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:27:14.283408 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:14.283285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-72skd" Jul 29 06:27:14.407208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:14.407179 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-72skd"] Jul 29 06:27:14.409608 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:27:14.409582 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9deaae_8679_4bd7_905f_d3457077a28b.slice/crio-4204a911e644042d14f72a582a965e25789ba48b4418e5f0d3ad4b0735cb0b5e WatchSource:0}: Error finding container 4204a911e644042d14f72a582a965e25789ba48b4418e5f0d3ad4b0735cb0b5e: Status 404 returned error can't find the container with id 4204a911e644042d14f72a582a965e25789ba48b4418e5f0d3ad4b0735cb0b5e Jul 29 06:27:14.828204 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:14.828115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72skd" event={"ID":"bc9deaae-8679-4bd7-905f-d3457077a28b","Type":"ContainerStarted","Data":"4204a911e644042d14f72a582a965e25789ba48b4418e5f0d3ad4b0735cb0b5e"} Jul 29 06:27:15.653917 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:15.653880 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:15.718615 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:15.718581 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:15.922474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:15.922243 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:18.494486 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:27:18.494450 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w9ltd" podUID="ce8b77c4-2fc8-4bd3-891d-94f96aca97d9" Jul 29 06:27:18.843449 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:18.843345 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:19.847718 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:19.847684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72skd" event={"ID":"bc9deaae-8679-4bd7-905f-d3457077a28b","Type":"ContainerStarted","Data":"d60f10acfd9f963114c2ca5b16f6a4ab3167e885d7605c7360ad7d0f73b32aa8"} Jul 29 06:27:20.319310 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:20.319272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:21.789889 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:21.789856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:21.915808 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:21.915772 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.021179 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.021096 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.056722 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.056645 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8dxgz"] Jul 29 06:27:22.072759 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.072733 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.075751 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.075727 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 06:27:22.076004 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.075983 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 06:27:22.076100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.076012 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-chckx\"" Jul 29 06:27:22.076150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.076136 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 06:27:22.076280 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.076265 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 06:27:22.076403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.076361 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 06:27:22.077470 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.077353 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 06:27:22.099912 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.099866 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.193529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193490 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193544 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-wtmp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193584 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-textfile\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193616 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-root\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193662 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntfp\" (UniqueName: \"kubernetes.io/projected/a896ceac-3ce1-45c5-ac59-850c78696ebe-kube-api-access-4ntfp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-metrics-client-ca\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193780 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-sys\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.193964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.193867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-tls\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.199426 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.199397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.294789 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294754 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-sys\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294824 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-tls\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294877 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-wtmp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294892 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-sys\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294903 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-textfile\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.294979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.294965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-root\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntfp\" (UniqueName: \"kubernetes.io/projected/a896ceac-3ce1-45c5-ac59-850c78696ebe-kube-api-access-4ntfp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-metrics-client-ca\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295178 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-textfile\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295278 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-wtmp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295652 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896ceac-3ce1-45c5-ac59-850c78696ebe-root\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295843 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-metrics-client-ca\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.295938 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.295905 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-accelerators-collector-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.297732 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.297696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce8b77c4-2fc8-4bd3-891d-94f96aca97d9-metrics-tls\") pod \"dns-default-w9ltd\" (UID: \"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9\") " pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:22.297979 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.297933 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-tls\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.298088 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.298052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896ceac-3ce1-45c5-ac59-850c78696ebe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.300289 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.300259 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.307935 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.307884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntfp\" (UniqueName: \"kubernetes.io/projected/a896ceac-3ce1-45c5-ac59-850c78696ebe-kube-api-access-4ntfp\") pod \"node-exporter-8dxgz\" (UID: \"a896ceac-3ce1-45c5-ac59-850c78696ebe\") " pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.383986 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.383955 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8dxgz" Jul 29 06:27:22.398009 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.397947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:22.446888 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.446848 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z5482\"" Jul 29 06:27:22.455466 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.455265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:22.584938 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:22.584863 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:23.014778 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.014746 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:23.097115 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.097045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:23.214569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.214539 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:23.272276 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:27:23.272251 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda896ceac_3ce1_45c5_ac59_850c78696ebe.slice/crio-082c0e51a2cb10c5f77dc614020e10c44921f1a8343acb4c02d7f4eeb4df2337 WatchSource:0}: Error finding container 082c0e51a2cb10c5f77dc614020e10c44921f1a8343acb4c02d7f4eeb4df2337: Status 404 returned error can't find the container with id 082c0e51a2cb10c5f77dc614020e10c44921f1a8343acb4c02d7f4eeb4df2337 Jul 29 06:27:23.387429 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.387407 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9ltd"] Jul 29 06:27:23.389832 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:27:23.389807 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8b77c4_2fc8_4bd3_891d_94f96aca97d9.slice/crio-0d2aa221c91bc3ba4bdf60536f0215d748b39b92c5d8375fe19da70ed1359ae1 WatchSource:0}: Error finding container 0d2aa221c91bc3ba4bdf60536f0215d748b39b92c5d8375fe19da70ed1359ae1: Status 404 returned error can't find the container with id 0d2aa221c91bc3ba4bdf60536f0215d748b39b92c5d8375fe19da70ed1359ae1 Jul 29 06:27:23.859944 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.859907 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9ltd" event={"ID":"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9","Type":"ContainerStarted","Data":"0d2aa221c91bc3ba4bdf60536f0215d748b39b92c5d8375fe19da70ed1359ae1"} Jul 29 06:27:23.861070 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.861050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dxgz" event={"ID":"a896ceac-3ce1-45c5-ac59-850c78696ebe","Type":"ContainerStarted","Data":"082c0e51a2cb10c5f77dc614020e10c44921f1a8343acb4c02d7f4eeb4df2337"} Jul 29 06:27:23.862675 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:23.862653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72skd" event={"ID":"bc9deaae-8679-4bd7-905f-d3457077a28b","Type":"ContainerStarted","Data":"7c0ab9662b813906d0034d53312031a166d3639952de5d902234b71f3e3bf1b6"} Jul 29 06:27:24.867971 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:24.867883 2556 generic.go:356] "Generic (PLEG): container finished" podID="a896ceac-3ce1-45c5-ac59-850c78696ebe" containerID="7c17e9d39e410a91130e063937759078cc0e51c363c32f9aff0b8ac863dbd2e4" exitCode=0 Jul 29 06:27:24.868367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:24.867963 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dxgz" event={"ID":"a896ceac-3ce1-45c5-ac59-850c78696ebe","Type":"ContainerDied","Data":"7c17e9d39e410a91130e063937759078cc0e51c363c32f9aff0b8ac863dbd2e4"} Jul 29 06:27:24.967788 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:24.967750 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:25.066609 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.066525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:25.218962 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.218911 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:25.728634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.728583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:25.874372 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.874337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dxgz" event={"ID":"a896ceac-3ce1-45c5-ac59-850c78696ebe","Type":"ContainerStarted","Data":"0b49fbdfe8b025b5df1a961177fd6f29e47f0a5cde11aad8014a1884fd3773e3"} Jul 29 06:27:25.874872 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.874429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8dxgz" event={"ID":"a896ceac-3ce1-45c5-ac59-850c78696ebe","Type":"ContainerStarted","Data":"9429a00b7c6bb815060b988ac22b35005d505d2f794b923b306734323ff4d733"} Jul 29 06:27:25.895095 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:25.895050 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8dxgz" podStartSLOduration=2.475982981 podStartE2EDuration="3.895034165s" totalImagesPullingTime="1.333476545s" totalInitContainerRuntime="85.574639ms" isStatefulPod=true podCreationTimestamp="2026-07-29 06:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:25.893195752 +0000 UTC m=+290.427275350" watchObservedRunningTime="2026-07-29 06:27:25.895034165 +0000 UTC m=+290.429113737" Jul 29 06:27:26.333199 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.333168 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fcc59db9d-x9mc5"] Jul 29 06:27:26.336892 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.336869 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.341057 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341005 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 06:27:26.341057 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341011 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mwkld\"" Jul 29 06:27:26.341237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341026 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eqgpf0pj5f3rp\"" Jul 29 06:27:26.341237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341078 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 06:27:26.341237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341011 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 06:27:26.341237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.341026 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 06:27:26.345853 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.345802 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcc59db9d-x9mc5"] Jul 29 06:27:26.425891 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.425856 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-client-certs\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.425891 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.425899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48744\" (UniqueName: \"kubernetes.io/projected/30860552-c0c5-4a08-b690-1bcab249879d-kube-api-access-48744\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.426131 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.425925 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-metrics-server-audit-profiles\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.426131 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.426025 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-client-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.426131 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.426075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.426131 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.426104 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30860552-c0c5-4a08-b690-1bcab249879d-audit-log\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.426407 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.426146 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-tls\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527541 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-client-certs\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527721 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48744\" (UniqueName: \"kubernetes.io/projected/30860552-c0c5-4a08-b690-1bcab249879d-kube-api-access-48744\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527721 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-metrics-server-audit-profiles\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527721 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-client-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527907 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.527907 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30860552-c0c5-4a08-b690-1bcab249879d-audit-log\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.528036 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.527924 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-tls\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.528395 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.528307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30860552-c0c5-4a08-b690-1bcab249879d-audit-log\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.528624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.528607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.528996 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.528971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30860552-c0c5-4a08-b690-1bcab249879d-metrics-server-audit-profiles\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.530550 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.530524 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-client-ca-bundle\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.530696 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.530676 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-tls\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.530801 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.530701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30860552-c0c5-4a08-b690-1bcab249879d-secret-metrics-server-client-certs\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.535424 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.535393 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48744\" (UniqueName: \"kubernetes.io/projected/30860552-c0c5-4a08-b690-1bcab249879d-kube-api-access-48744\") pod \"metrics-server-7fcc59db9d-x9mc5\" (UID: \"30860552-c0c5-4a08-b690-1bcab249879d\") " pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.595699 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.595614 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:26.595699 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.595665 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:26.648394 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.648334 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:26.802614 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:26.802561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:27.358845 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.358817 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcc59db9d-x9mc5"] Jul 29 06:27:27.361512 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:27:27.361486 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30860552_c0c5_4a08_b690_1bcab249879d.slice/crio-622547fafdb2cf424b78f7d23a994306ef4e755e1e147f68e930133601604d32 WatchSource:0}: Error finding container 622547fafdb2cf424b78f7d23a994306ef4e755e1e147f68e930133601604d32: Status 404 returned error can't find the container with id 622547fafdb2cf424b78f7d23a994306ef4e755e1e147f68e930133601604d32 Jul 29 06:27:27.373014 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.372978 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:27.807769 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.807733 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:27.807918 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.807790 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:27.883004 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.882968 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9ltd" event={"ID":"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9","Type":"ContainerStarted","Data":"2f9e9430740451f679044313c91975f312707a7f0d72d9fa7b91fc5e4161311f"} Jul 29 06:27:27.883195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.883043 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:27.883195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.883072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9ltd" event={"ID":"ce8b77c4-2fc8-4bd3-891d-94f96aca97d9","Type":"ContainerStarted","Data":"ce025472ddb131283f2039c66418301691f6f68888940eabb71b2dce7659460a"} Jul 29 06:27:27.885250 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.885220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-72skd" event={"ID":"bc9deaae-8679-4bd7-905f-d3457077a28b","Type":"ContainerStarted","Data":"0f0b9ea098ceebb8218a621b40416c47f3fb42334ddc2846d75eb7cdc0fc9f75"} Jul 29 06:27:27.886572 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.886546 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" event={"ID":"30860552-c0c5-4a08-b690-1bcab249879d","Type":"ContainerStarted","Data":"622547fafdb2cf424b78f7d23a994306ef4e755e1e147f68e930133601604d32"} Jul 29 06:27:27.898643 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.898592 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w9ltd" podStartSLOduration=252.06434571 podStartE2EDuration="4m15.898575296s" totalImagesPullingTime="3.834229586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:27.898023392 +0000 UTC m=+292.432102966" watchObservedRunningTime="2026-07-29 06:27:27.898575296 +0000 UTC m=+292.432654870" Jul 29 06:27:27.919454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:27.919406 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-72skd" podStartSLOduration=33.215138208 podStartE2EDuration="45.919369892s" totalImagesPullingTime="12.704231684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:26:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:27.919196609 +0000 UTC m=+292.453276183" watchObservedRunningTime="2026-07-29 06:27:27.919369892 +0000 UTC m=+292.453449469" Jul 29 06:27:28.243045 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:28.243006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:28.891772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:28.891678 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" event={"ID":"30860552-c0c5-4a08-b690-1bcab249879d","Type":"ContainerStarted","Data":"17d09fed2a1f41891e0eca45656087408b568455d2300701719a5a95f669dcb2"} Jul 29 06:27:28.908982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:28.908940 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" podStartSLOduration=1.651214479 podStartE2EDuration="2.908926505s" totalImagesPullingTime="1.257712026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:28.908295084 +0000 UTC m=+293.442374658" watchObservedRunningTime="2026-07-29 06:27:28.908926505 +0000 UTC m=+293.443006077" Jul 29 06:27:29.454014 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:29.453977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:29.554179 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:29.554145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:31.923517 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:27:31.923477 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dqvg9" podUID="e17be3de-940a-4138-8d18-94f587b8e655" Jul 29 06:27:33.712022 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:33.711946 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:33.808452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:33.808412 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:33.858213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:33.858184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.193663 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.193628 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.377169 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.377134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.427654 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.427624 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.586313 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.586235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.801196 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.801158 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:34.850481 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:34.850412 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:35.188898 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:35.188826 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:35.239990 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:35.239939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:35.414290 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:35.414249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:35.789988 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:35.789914 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:35.803832 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:35.803811 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 06:27:36.595351 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:36.595316 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:36.597470 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:36.595365 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:37.595100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:37.595063 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:37.643931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:37.643887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:37.807547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:37.807513 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:37.807718 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:37.807563 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:37.894462 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:37.894367 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w9ltd" Jul 29 06:27:41.104215 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:41.104184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:41.206513 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:41.206475 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:43.804873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:43.804836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:43.903578 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:43.903548 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:43.922323 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:43.922303 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:27:45.372438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:45.372396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:45.473870 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:45.473839 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:46.210873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.210837 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:46.309086 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.309047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:46.595490 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.595411 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:46.595490 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.595460 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:46.595893 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.595502 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:46.595956 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.595920 2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"0a6dd862e52e36390631a7e75a06bc1baa15adf18bb8726970a230bac9766e8e"} pod="openshift-image-registry/image-registry-566654d898-kc6wh" containerMessage="Container registry failed liveness probe, will be restarted" Jul 29 06:27:46.599680 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.599654 2556 patch_prober.go:28] interesting pod/image-registry-566654d898-kc6wh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 06:27:46.599814 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.599708 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:27:46.649464 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.649434 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:46.649610 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:46.649487 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:27:47.029556 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:47.029524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:47.129580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:47.129547 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:51.204034 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:51.204003 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:51.261239 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:51.261210 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:52.469800 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:52.469766 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:52.520151 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:52.520100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:55.802021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:55.801977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:55.902391 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:55.902343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:56.009530 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.009499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:56.062073 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.061995 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:56.600131 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.600106 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:27:56.723793 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.723755 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:56.842526 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.842495 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:56.893022 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:56.892907 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:59.615620 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:59.615586 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:27:59.660397 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:27:59.660347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:06.653809 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:06.653783 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:28:06.661791 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:06.661766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fcc59db9d-x9mc5" Jul 29 06:28:07.477304 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:07.477267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:07.580686 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:07.580652 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:08.683454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:08.683424 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:08.734759 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:08.734731 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:09.014163 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:09.014131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:09.065314 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:09.065254 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:10.587468 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:10.587430 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" podUID="95d2becb-2b70-4e3c-a899-41ae81e2f5af" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 06:28:11.614343 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:11.614304 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-566654d898-kc6wh" podUID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerName="registry" containerID="cri-o://0a6dd862e52e36390631a7e75a06bc1baa15adf18bb8726970a230bac9766e8e" gracePeriod=30 Jul 29 06:28:13.026966 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.026928 2556 generic.go:356] "Generic (PLEG): container finished" podID="12733955-7a4c-4710-8cf2-f610aa9bcd8f" containerID="0a6dd862e52e36390631a7e75a06bc1baa15adf18bb8726970a230bac9766e8e" exitCode=0 Jul 29 06:28:13.027335 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.027014 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566654d898-kc6wh" event={"ID":"12733955-7a4c-4710-8cf2-f610aa9bcd8f","Type":"ContainerDied","Data":"0a6dd862e52e36390631a7e75a06bc1baa15adf18bb8726970a230bac9766e8e"} Jul 29 06:28:13.027335 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.027055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566654d898-kc6wh" event={"ID":"12733955-7a4c-4710-8cf2-f610aa9bcd8f","Type":"ContainerStarted","Data":"6ca39929d9cabb80b0b321dd0c3692b46d59e539a1a4f5e4d04f30f39b173665"} Jul 29 06:28:13.027335 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.027101 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:28:13.100777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.100748 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs5xp_df1d2617-2492-4ea0-b258-c2a27e18d731/serve-healthcheck-canary/0.log" Jul 29 06:28:13.352169 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.352066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:13.453067 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:13.453037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:20.586887 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:20.586845 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" podUID="95d2becb-2b70-4e3c-a899-41ae81e2f5af" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 06:28:22.474764 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:22.474724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:22.525457 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:22.525429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:29.701710 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:29.701623 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:29.754827 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:29.754792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:30.586838 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:30.586799 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" podUID="95d2becb-2b70-4e3c-a899-41ae81e2f5af" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 29 06:28:30.587035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:30.586878 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" Jul 29 06:28:30.587449 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:30.587430 2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b0b1f00da888588ff01c9c79ab78b2ea72e741eac67590a6057b6d69ccd68043"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 29 06:28:30.587506 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:30.587473 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" podUID="95d2becb-2b70-4e3c-a899-41ae81e2f5af" containerName="service-proxy" containerID="cri-o://b0b1f00da888588ff01c9c79ab78b2ea72e741eac67590a6057b6d69ccd68043" gracePeriod=30 Jul 29 06:28:31.085643 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:31.085607 2556 generic.go:356] "Generic (PLEG): container finished" podID="95d2becb-2b70-4e3c-a899-41ae81e2f5af" containerID="b0b1f00da888588ff01c9c79ab78b2ea72e741eac67590a6057b6d69ccd68043" exitCode=2 Jul 29 06:28:31.086097 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:31.085683 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerDied","Data":"b0b1f00da888588ff01c9c79ab78b2ea72e741eac67590a6057b6d69ccd68043"} Jul 29 06:28:31.086097 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:31.085738 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84f77b8d5c-xwlt9" event={"ID":"95d2becb-2b70-4e3c-a899-41ae81e2f5af","Type":"ContainerStarted","Data":"a615f47b99736240b182d49fef6ca673ce13a806d05880890571377b80071ff8"} Jul 29 06:28:34.035816 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:34.035777 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-566654d898-kc6wh" Jul 29 06:28:34.286304 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:34.286213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:34.335494 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:34.335463 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:41.218579 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:41.218541 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:41.458366 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:41.458333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:46.309358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.309326 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f96b55844-g6snr"] Jul 29 06:28:46.312639 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.312623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.315316 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.315293 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 06:28:46.315316 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.315306 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 06:28:46.316368 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.316342 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 06:28:46.316590 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.316494 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 06:28:46.316680 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.316654 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 06:28:46.316878 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.316853 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4kwtb\"" Jul 29 06:28:46.324555 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.324532 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 06:28:46.324665 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.324607 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f96b55844-g6snr"] Jul 29 06:28:46.409069 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409200 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409200 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409200 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-federate-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409244 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-serving-certs-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-metrics-client-ca\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.409421 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.409312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxzb\" (UniqueName: \"kubernetes.io/projected/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-kube-api-access-jqxzb\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510321 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-metrics-client-ca\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510321 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxzb\" (UniqueName: \"kubernetes.io/projected/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-kube-api-access-jqxzb\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510347 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-federate-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.510535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.510494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-serving-certs-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.511222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.511202 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-serving-certs-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.511398 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.511350 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-metrics-client-ca\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.511643 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.511617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.513188 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.513162 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.513269 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.513165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-telemeter-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.513514 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.513499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-federate-client-tls\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.513614 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.513588 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.517798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.517778 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxzb\" (UniqueName: \"kubernetes.io/projected/9f8b113d-073c-4eb0-aedc-1f77f1e496eb-kube-api-access-jqxzb\") pod \"telemeter-client-6f96b55844-g6snr\" (UID: \"9f8b113d-073c-4eb0-aedc-1f77f1e496eb\") " pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.624506 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.624442 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" Jul 29 06:28:46.752441 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.752415 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f96b55844-g6snr"] Jul 29 06:28:46.753680 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:28:46.753653 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8b113d_073c_4eb0_aedc_1f77f1e496eb.slice/crio-0b80aa7d3bd1aaec64d9dbbceab586f4631263e9149a40f2f1f013083b42cb40 WatchSource:0}: Error finding container 0b80aa7d3bd1aaec64d9dbbceab586f4631263e9149a40f2f1f013083b42cb40: Status 404 returned error can't find the container with id 0b80aa7d3bd1aaec64d9dbbceab586f4631263e9149a40f2f1f013083b42cb40 Jul 29 06:28:46.755653 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:46.755636 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 06:28:47.141273 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:47.141238 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" event={"ID":"9f8b113d-073c-4eb0-aedc-1f77f1e496eb","Type":"ContainerStarted","Data":"0b80aa7d3bd1aaec64d9dbbceab586f4631263e9149a40f2f1f013083b42cb40"} Jul 29 06:28:49.150112 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:49.150029 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" event={"ID":"9f8b113d-073c-4eb0-aedc-1f77f1e496eb","Type":"ContainerStarted","Data":"658ecb8f392706ba7941044f2e34a7704a9bec2cb641a9abad8f04b44b864812"} Jul 29 06:28:49.834121 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:49.834085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:28:49.836500 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:49.836477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17be3de-940a-4138-8d18-94f587b8e655-metrics-certs\") pod \"network-metrics-daemon-dqvg9\" (UID: \"e17be3de-940a-4138-8d18-94f587b8e655\") " pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:28:49.931397 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:49.929194 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btmpr\"" Jul 29 06:28:49.937508 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:49.937484 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqvg9" Jul 29 06:28:50.074208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:50.074185 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqvg9"] Jul 29 06:28:50.075793 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:28:50.075756 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17be3de_940a_4138_8d18_94f587b8e655.slice/crio-aaed0e39620940119182fb9d477200fbfbd606c661228adb1750de98201e9470 WatchSource:0}: Error finding container aaed0e39620940119182fb9d477200fbfbd606c661228adb1750de98201e9470: Status 404 returned error can't find the container with id aaed0e39620940119182fb9d477200fbfbd606c661228adb1750de98201e9470 Jul 29 06:28:50.157493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:50.157370 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" event={"ID":"9f8b113d-073c-4eb0-aedc-1f77f1e496eb","Type":"ContainerStarted","Data":"8a715a5650c321752a033b314b1b6b95940f32daf6cdaaf857479ff0e7e1e8aa"} Jul 29 06:28:50.157985 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:50.157504 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" event={"ID":"9f8b113d-073c-4eb0-aedc-1f77f1e496eb","Type":"ContainerStarted","Data":"a6ef9ec765103eeb2b7a4c734e0a34aa1d3dbc215cde97f852301e90a27c3cf5"} Jul 29 06:28:50.159164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:50.159140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqvg9" event={"ID":"e17be3de-940a-4138-8d18-94f587b8e655","Type":"ContainerStarted","Data":"aaed0e39620940119182fb9d477200fbfbd606c661228adb1750de98201e9470"} Jul 29 06:28:50.179290 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:50.179246 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f96b55844-g6snr" podStartSLOduration=1.13738774 podStartE2EDuration="4.179232506s" totalImagesPullingTime="3.041844766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:28:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:28:50.177703696 +0000 UTC m=+374.711783269" watchObservedRunningTime="2026-07-29 06:28:50.179232506 +0000 UTC m=+374.713312079" Jul 29 06:28:51.167322 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.167290 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqvg9" event={"ID":"e17be3de-940a-4138-8d18-94f587b8e655","Type":"ContainerStarted","Data":"af9a2e13224489b96720bc51455b9f5f5d908ad005b7bec58effd567fea000b9"} Jul 29 06:28:51.167691 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.167351 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqvg9" event={"ID":"e17be3de-940a-4138-8d18-94f587b8e655","Type":"ContainerStarted","Data":"77809d804bead4c3981671ee51e77138ec6f450e409431f3acab2d203af8586d"} Jul 29 06:28:51.184973 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.184928 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dqvg9" podStartSLOduration=373.254491141 podStartE2EDuration="6m14.18491441s" totalImagesPullingTime="930.423269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:28:51.184853265 +0000 UTC m=+375.718932841" watchObservedRunningTime="2026-07-29 06:28:51.18491441 +0000 UTC m=+375.718993983" Jul 29 06:28:51.206795 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.206770 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:51.307696 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.307616 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:51.358183 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.358147 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:51.468822 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.468793 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:51.548484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:51.548450 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:52.066426 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:52.066391 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:52.120540 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:52.120505 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:52.752668 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:52.752636 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:52.808435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:52.808401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:54.293577 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:54.293546 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:28:54.345683 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:28:54.345647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:02.371077 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:02.371036 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:02.481070 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:02.481025 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:02.525837 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:02.525805 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:07.477549 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:07.477513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:07.527443 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:07.527408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:43.124100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:43.124062 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:56.437443 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:56.437326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:29:56.495073 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:29:56.495045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:08.366493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:08.366459 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:08.417202 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:08.417168 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:11.975269 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:11.975230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:22.484832 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:22.484789 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:22.583345 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:22.583310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:23.170942 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:23.170909 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:33.793747 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:33.793713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:33.894340 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:33.894310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:44.741236 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:44.741194 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:44.786233 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:44.786167 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:30:44.859410 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:30:44.859360 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:14.100360 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:14.100318 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:14.199500 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:14.199466 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:20.433059 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:20.433021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:40.455921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:40.455842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:41.580528 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:41.580488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:41.825420 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:41.825366 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:44.069368 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:44.069330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:44.170417 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:44.170367 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:45.357657 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:45.357627 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:45.454099 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:45.454064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:46.437683 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:46.437647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:46.539005 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:46.538967 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:48.444922 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:48.444867 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:48.828142 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:48.828068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:50.858091 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:50.858045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:50.912901 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:50.912871 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:57.075917 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:57.075884 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:57.171770 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:57.171736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:57.332778 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:57.332683 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:57.394278 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:57.394245 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:31:57.645777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:31:57.645689 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:02.657408 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:02.657339 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:02.757840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:02.757806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:04.907659 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:04.907629 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:21.670369 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:21.670332 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:21.770480 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:21.770426 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:22.479955 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.479919 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:22.679121 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.679083 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:22.772317 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.772219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:22.800734 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.800702 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb"] Jul 29 06:32:22.803797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.803780 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:22.806697 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.806672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdzx8\"" Jul 29 06:32:22.806697 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.806688 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 06:32:22.806866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.806709 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 06:32:22.806866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.806759 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 06:32:22.807074 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.807056 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 06:32:22.816331 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.816309 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb"] Jul 29 06:32:22.930688 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.930643 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:22.959731 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.959699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4s54\" (UniqueName: \"kubernetes.io/projected/05fcf4a5-b597-448f-b6a8-7163b05fdc75-kube-api-access-b4s54\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:22.959833 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.959750 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:22.959833 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.959805 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:22.959905 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.959846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:22.959905 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:22.959871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.038073 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.037992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:23.060618 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.060591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.060716 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.060629 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.060716 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.060665 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.060857 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.060835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.060911 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.060899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4s54\" (UniqueName: \"kubernetes.io/projected/05fcf4a5-b597-448f-b6a8-7163b05fdc75-kube-api-access-b4s54\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.061085 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.061068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.063043 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.063019 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05fcf4a5-b597-448f-b6a8-7163b05fdc75-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.063288 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.063267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.063357 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.063278 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05fcf4a5-b597-448f-b6a8-7163b05fdc75-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.076722 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.076699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4s54\" (UniqueName: \"kubernetes.io/projected/05fcf4a5-b597-448f-b6a8-7163b05fdc75-kube-api-access-b4s54\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-fvrsb\" (UID: \"05fcf4a5-b597-448f-b6a8-7163b05fdc75\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.113307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.113266 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:23.133861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.133828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:23.236264 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.236229 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:23.253993 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.253970 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb"] Jul 29 06:32:23.256456 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:32:23.256432 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fcf4a5_b597_448f_b6a8_7163b05fdc75.slice/crio-edd354c2820acc31bf96ee45d74273b2cab7ce7f3abe02d195cbd66c6a60c986 WatchSource:0}: Error finding container edd354c2820acc31bf96ee45d74273b2cab7ce7f3abe02d195cbd66c6a60c986: Status 404 returned error can't find the container with id edd354c2820acc31bf96ee45d74273b2cab7ce7f3abe02d195cbd66c6a60c986 Jul 29 06:32:23.336946 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.336861 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:23.891601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:23.891560 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" event={"ID":"05fcf4a5-b597-448f-b6a8-7163b05fdc75","Type":"ContainerStarted","Data":"edd354c2820acc31bf96ee45d74273b2cab7ce7f3abe02d195cbd66c6a60c986"} Jul 29 06:32:25.794029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:25.793990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:26.904522 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:26.904484 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" event={"ID":"05fcf4a5-b597-448f-b6a8-7163b05fdc75","Type":"ContainerStarted","Data":"1a43ddc0f631d4461725e6837d42bc3825c2dde2fbe89edf772d7b3f892b24a4"} Jul 29 06:32:27.909626 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:27.909533 2556 generic.go:356] "Generic (PLEG): container finished" podID="05fcf4a5-b597-448f-b6a8-7163b05fdc75" containerID="1a43ddc0f631d4461725e6837d42bc3825c2dde2fbe89edf772d7b3f892b24a4" exitCode=0 Jul 29 06:32:27.909626 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:27.909603 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" event={"ID":"05fcf4a5-b597-448f-b6a8-7163b05fdc75","Type":"ContainerDied","Data":"1a43ddc0f631d4461725e6837d42bc3825c2dde2fbe89edf772d7b3f892b24a4"} Jul 29 06:32:28.915295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:28.915203 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" event={"ID":"05fcf4a5-b597-448f-b6a8-7163b05fdc75","Type":"ContainerStarted","Data":"e7f2ef437c13ddb427c82c9dcc6f55729282f90a187710fc439d28e84e103ae7"} Jul 29 06:32:28.915723 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:28.915344 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:28.935246 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:28.935188 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" podStartSLOduration=2.160405805 podStartE2EDuration="6.935172693s" totalImagesPullingTime="3.703528s" totalInitContainerRuntime="1.071238888s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:28.934608609 +0000 UTC m=+593.468688179" watchObservedRunningTime="2026-07-29 06:32:28.935172693 +0000 UTC m=+593.469252266" Jul 29 06:32:34.678087 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:34.678042 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:34.779745 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:34.779708 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:39.605484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.605443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:39.677882 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.677848 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm"] Jul 29 06:32:39.679934 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.679917 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.682795 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.682773 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 06:32:39.684101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.684084 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 06:32:39.684453 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.684427 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 06:32:39.684580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.684455 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 06:32:39.684580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.684512 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 06:32:39.684689 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.684585 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8wnsf\"" Jul 29 06:32:39.700208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.700180 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm"] Jul 29 06:32:39.700467 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.700441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:39.789242 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.789208 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92vw\" (UniqueName: \"kubernetes.io/projected/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-kube-api-access-s92vw\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.789472 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.789289 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-manager-config\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.789472 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.789316 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.789472 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.789351 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.890260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.890172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92vw\" (UniqueName: \"kubernetes.io/projected/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-kube-api-access-s92vw\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.890260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.890234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-manager-config\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.890260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.890255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.890547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.890280 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.891026 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.890995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-manager-config\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.892896 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.892878 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.892990 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.892897 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-cert\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.905624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.905605 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92vw\" (UniqueName: \"kubernetes.io/projected/4d4d2a47-6657-4fb3-8b20-f05d87c29b59-kube-api-access-s92vw\") pod \"lws-controller-manager-68c9b6dddc-4vjgm\" (UID: \"4d4d2a47-6657-4fb3-8b20-f05d87c29b59\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:39.921125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.921101 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-fvrsb" Jul 29 06:32:39.988560 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:39.988528 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:40.112631 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:40.112607 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm"] Jul 29 06:32:40.114159 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:32:40.114127 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4d2a47_6657_4fb3_8b20_f05d87c29b59.slice/crio-ed41f816333bcd8b656408c4f7603ae1194b31d5754fde670aae56bdaaa61635 WatchSource:0}: Error finding container ed41f816333bcd8b656408c4f7603ae1194b31d5754fde670aae56bdaaa61635: Status 404 returned error can't find the container with id ed41f816333bcd8b656408c4f7603ae1194b31d5754fde670aae56bdaaa61635 Jul 29 06:32:40.957160 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:40.957124 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" event={"ID":"4d4d2a47-6657-4fb3-8b20-f05d87c29b59","Type":"ContainerStarted","Data":"ed41f816333bcd8b656408c4f7603ae1194b31d5754fde670aae56bdaaa61635"} Jul 29 06:32:41.699162 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:41.699125 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:41.808486 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:41.808445 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:42.716454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.716421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:42.774314 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.774280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:42.884069 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.884031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:42.965483 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.965449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" event={"ID":"4d4d2a47-6657-4fb3-8b20-f05d87c29b59","Type":"ContainerStarted","Data":"86d66aa6a6964b6b9e4b3c6833bf5668eed1c33be09796067507901cff972a3c"} Jul 29 06:32:42.965636 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.965548 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:42.977906 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.977840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:42.982860 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:42.982823 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" podStartSLOduration=1.893636331 podStartE2EDuration="3.982810642s" totalImagesPullingTime="2.089174311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:42.981617436 +0000 UTC m=+607.515697023" watchObservedRunningTime="2026-07-29 06:32:42.982810642 +0000 UTC m=+607.516890215" Jul 29 06:32:43.310834 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:43.310751 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:43.407396 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:43.407336 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:43.808629 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:43.808594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:51.276256 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:51.276220 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:51.363730 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:51.363684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:51.487728 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:51.487694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:51.606632 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:51.606556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:52.521359 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:52.521332 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:52.626572 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:52.626530 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:52.722165 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:52.722128 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:52.903873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:52.903782 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:53.404331 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:53.404292 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:53.970569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:53.970539 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-4vjgm" Jul 29 06:32:54.469140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:54.469110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:54.534038 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:54.534006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:55.420916 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:55.420874 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:58.885256 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:58.885172 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:58.949059 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:58.949028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:59.358171 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.358141 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt"] Jul 29 06:32:59.360053 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.360037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.362751 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.362724 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zclm9\"" Jul 29 06:32:59.362751 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.362724 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 06:32:59.362924 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.362769 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 06:32:59.363017 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.363001 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 06:32:59.372584 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.372565 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt"] Jul 29 06:32:59.387280 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.387253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:59.422358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422332 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422369 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422570 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422570 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422529 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422570 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422579 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qqn\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-kube-api-access-k7qqn\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422605 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.422667 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.422628 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88979fb6-2d23-4cf3-ae48-c41f48e23238-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.523935 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.523905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.523935 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.523937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524127 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.523966 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524127 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.523995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524127 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524116 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qqn\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-kube-api-access-k7qqn\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524220 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524244 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88979fb6-2d23-4cf3-ae48-c41f48e23238-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524494 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524522 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524564 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.524601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.524580 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.525331 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.525307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88979fb6-2d23-4cf3-ae48-c41f48e23238-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.526554 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.526538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.526792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.526771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.546025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.545996 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qqn\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-kube-api-access-k7qqn\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.547889 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.547867 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88979fb6-2d23-4cf3-ae48-c41f48e23238-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cn74bt\" (UID: \"88979fb6-2d23-4cf3-ae48-c41f48e23238\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.647926 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.647854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:32:59.670150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.670130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:32:59.797472 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.797444 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt"] Jul 29 06:32:59.799070 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:32:59.799041 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88979fb6_2d23_4cf3_ae48_c41f48e23238.slice/crio-b13fc9e34d39ae8504f88ba6bc2dd4679112c8252c3cdd61ff3fecd450fab7de WatchSource:0}: Error finding container b13fc9e34d39ae8504f88ba6bc2dd4679112c8252c3cdd61ff3fecd450fab7de: Status 404 returned error can't find the container with id b13fc9e34d39ae8504f88ba6bc2dd4679112c8252c3cdd61ff3fecd450fab7de Jul 29 06:32:59.968128 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:32:59.968091 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:00.022754 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:00.022724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" event={"ID":"88979fb6-2d23-4cf3-ae48-c41f48e23238","Type":"ContainerStarted","Data":"b13fc9e34d39ae8504f88ba6bc2dd4679112c8252c3cdd61ff3fecd450fab7de"} Jul 29 06:33:02.587924 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:02.587883 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:33:02.588208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:02.587980 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:33:02.588208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:02.588008 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:33:03.043372 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:03.043334 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" event={"ID":"88979fb6-2d23-4cf3-ae48-c41f48e23238","Type":"ContainerStarted","Data":"9bcee6eebdbc3de66f948144355da3930b47f4e097e0e44a8bf7527d5886ede0"} Jul 29 06:33:03.063427 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:03.063360 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" podStartSLOduration=1.276777435 podStartE2EDuration="4.063344932s" totalImagesPullingTime="2.786567497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:33:03.061244822 +0000 UTC m=+627.595324394" watchObservedRunningTime="2026-07-29 06:33:03.063344932 +0000 UTC m=+627.597424505" Jul 29 06:33:03.670372 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:03.670339 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:33:03.674898 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:03.674876 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:33:04.047156 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:04.047126 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:33:04.048141 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:04.048120 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cn74bt" Jul 29 06:33:36.736759 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:36.736724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:38.669042 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:38.669009 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:38.717248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:38.717216 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:43.256810 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.256772 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:43.319151 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.319119 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:43.445777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.445749 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:43.454408 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.454370 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:43.458994 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.458968 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 06:33:43.459735 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.459716 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 06:33:43.459849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.459722 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-m4rrf\"" Jul 29 06:33:43.464309 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.464286 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:43.497675 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.497649 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:43.549778 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.549697 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmk8\" (UniqueName: \"kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8\") pod \"kuadrant-operator-catalog-z24d9\" (UID: \"9550346a-b72e-4a46-953b-1f546569d087\") " pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:43.650878 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.650846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmk8\" (UniqueName: \"kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8\") pod \"kuadrant-operator-catalog-z24d9\" (UID: \"9550346a-b72e-4a46-953b-1f546569d087\") " pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:43.658959 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.658928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmk8\" (UniqueName: \"kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8\") pod \"kuadrant-operator-catalog-z24d9\" (UID: \"9550346a-b72e-4a46-953b-1f546569d087\") " pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:43.765432 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.765358 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:43.821426 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.821393 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:43.891037 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:43.891005 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:43.893961 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:33:43.893934 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9550346a_b72e_4a46_953b_1f546569d087.slice/crio-6d5d915fceb6e5ca207fd233354830c95026ed356ac1ab6ac9b7d0abee3be72d WatchSource:0}: Error finding container 6d5d915fceb6e5ca207fd233354830c95026ed356ac1ab6ac9b7d0abee3be72d: Status 404 returned error can't find the container with id 6d5d915fceb6e5ca207fd233354830c95026ed356ac1ab6ac9b7d0abee3be72d Jul 29 06:33:44.030118 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.030081 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lnsgw"] Jul 29 06:33:44.034561 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.034538 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:44.040527 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.040499 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lnsgw"] Jul 29 06:33:44.153874 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.153797 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqcn\" (UniqueName: \"kubernetes.io/projected/e8722a78-6392-4173-a236-eaba28aeaa4d-kube-api-access-cnqcn\") pod \"kuadrant-operator-catalog-lnsgw\" (UID: \"e8722a78-6392-4173-a236-eaba28aeaa4d\") " pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:44.186669 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.186640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" event={"ID":"9550346a-b72e-4a46-953b-1f546569d087","Type":"ContainerStarted","Data":"6d5d915fceb6e5ca207fd233354830c95026ed356ac1ab6ac9b7d0abee3be72d"} Jul 29 06:33:44.254320 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.254278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqcn\" (UniqueName: \"kubernetes.io/projected/e8722a78-6392-4173-a236-eaba28aeaa4d-kube-api-access-cnqcn\") pod \"kuadrant-operator-catalog-lnsgw\" (UID: \"e8722a78-6392-4173-a236-eaba28aeaa4d\") " pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:44.263952 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.263928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqcn\" (UniqueName: \"kubernetes.io/projected/e8722a78-6392-4173-a236-eaba28aeaa4d-kube-api-access-cnqcn\") pod \"kuadrant-operator-catalog-lnsgw\" (UID: \"e8722a78-6392-4173-a236-eaba28aeaa4d\") " pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:44.344955 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.344925 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:44.477075 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:44.477043 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lnsgw"] Jul 29 06:33:44.504661 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:33:44.504630 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8722a78_6392_4173_a236_eaba28aeaa4d.slice/crio-d402e02bac85f43ecf4ddbeaabd8d5068d79deecc74f275e129494fc2c83f53e WatchSource:0}: Error finding container d402e02bac85f43ecf4ddbeaabd8d5068d79deecc74f275e129494fc2c83f53e: Status 404 returned error can't find the container with id d402e02bac85f43ecf4ddbeaabd8d5068d79deecc74f275e129494fc2c83f53e Jul 29 06:33:45.193212 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:45.193175 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" event={"ID":"e8722a78-6392-4173-a236-eaba28aeaa4d","Type":"ContainerStarted","Data":"d402e02bac85f43ecf4ddbeaabd8d5068d79deecc74f275e129494fc2c83f53e"} Jul 29 06:33:46.204681 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.204579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" event={"ID":"e8722a78-6392-4173-a236-eaba28aeaa4d","Type":"ContainerStarted","Data":"86430335e6ef1f7eefa02b59953cb3705cc9070c652eab15b1bc7b8beba0dee1"} Jul 29 06:33:46.206198 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.206173 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" event={"ID":"9550346a-b72e-4a46-953b-1f546569d087","Type":"ContainerStarted","Data":"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98"} Jul 29 06:33:46.206310 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.206265 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" podUID="9550346a-b72e-4a46-953b-1f546569d087" containerName="registry-server" containerID="cri-o://0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98" gracePeriod=2 Jul 29 06:33:46.220525 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.220485 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" podStartSLOduration=0.646490253 podStartE2EDuration="2.220468575s" totalImagesPullingTime="1.573978322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:33:46.219004237 +0000 UTC m=+670.753083822" watchObservedRunningTime="2026-07-29 06:33:46.220468575 +0000 UTC m=+670.754548148" Jul 29 06:33:46.233869 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.233825 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" podStartSLOduration=1.177632394 podStartE2EDuration="3.233811214s" totalImagesPullingTime="2.05617882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:33:46.232659529 +0000 UTC m=+670.766739124" watchObservedRunningTime="2026-07-29 06:33:46.233811214 +0000 UTC m=+670.767890787" Jul 29 06:33:46.443586 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.443565 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:46.572945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.572878 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbmk8\" (UniqueName: \"kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8\") pod \"9550346a-b72e-4a46-953b-1f546569d087\" (UID: \"9550346a-b72e-4a46-953b-1f546569d087\") " Jul 29 06:33:46.575082 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.575062 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8" (OuterVolumeSpecName: "kube-api-access-jbmk8") pod "9550346a-b72e-4a46-953b-1f546569d087" (UID: "9550346a-b72e-4a46-953b-1f546569d087"). InnerVolumeSpecName "kube-api-access-jbmk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:33:46.673399 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:46.673353 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmk8\" (UniqueName: \"kubernetes.io/projected/9550346a-b72e-4a46-953b-1f546569d087-kube-api-access-jbmk8\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:33:47.211545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.211509 2556 generic.go:356] "Generic (PLEG): container finished" podID="9550346a-b72e-4a46-953b-1f546569d087" containerID="0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98" exitCode=0 Jul 29 06:33:47.212010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.211557 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" Jul 29 06:33:47.212010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.211586 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" event={"ID":"9550346a-b72e-4a46-953b-1f546569d087","Type":"ContainerDied","Data":"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98"} Jul 29 06:33:47.212010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.211632 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z24d9" event={"ID":"9550346a-b72e-4a46-953b-1f546569d087","Type":"ContainerDied","Data":"6d5d915fceb6e5ca207fd233354830c95026ed356ac1ab6ac9b7d0abee3be72d"} Jul 29 06:33:47.212010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.211650 2556 scope.go:117] "RemoveContainer" containerID="0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98" Jul 29 06:33:47.220809 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.220786 2556 scope.go:117] "RemoveContainer" containerID="0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98" Jul 29 06:33:47.221058 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:33:47.221037 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98\": container with ID starting with 0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98 not found: ID does not exist" containerID="0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98" Jul 29 06:33:47.221105 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.221066 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98"} err="failed to get container status \"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98\": rpc error: code = NotFound desc = could not find container \"0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98\": container with ID starting with 0fa227c45722a66c8b47e9045abca96aa755e97c1f60709637e0006503b9cf98 not found: ID does not exist" Jul 29 06:33:47.232232 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.232180 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:47.234122 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.234102 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z24d9"] Jul 29 06:33:47.926134 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:47.926102 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9550346a-b72e-4a46-953b-1f546569d087" path="/var/lib/kubelet/pods/9550346a-b72e-4a46-953b-1f546569d087/volumes" Jul 29 06:33:51.128044 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:51.128004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:51.174288 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:51.174257 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:33:54.345763 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:54.345731 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:54.346154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:54.345786 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:54.366998 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:54.366967 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:33:55.267133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:33:55.267107 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lnsgw" Jul 29 06:34:01.823263 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:01.823225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:01.875207 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:01.875174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:01.922839 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:01.922808 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:11.758608 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:11.758556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:12.339751 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:12.339710 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:15.592905 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.592870 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-895pz"] Jul 29 06:34:15.593348 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.593151 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9550346a-b72e-4a46-953b-1f546569d087" containerName="registry-server" Jul 29 06:34:15.593348 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.593162 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9550346a-b72e-4a46-953b-1f546569d087" containerName="registry-server" Jul 29 06:34:15.593348 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.593219 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9550346a-b72e-4a46-953b-1f546569d087" containerName="registry-server" Jul 29 06:34:15.600097 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.600082 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:15.602849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.602833 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bm89f\"" Jul 29 06:34:15.608213 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.608189 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-895pz"] Jul 29 06:34:15.693735 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.693706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjzz\" (UniqueName: \"kubernetes.io/projected/cc14b502-a312-4b50-a82d-3a9c3c158d91-kube-api-access-9bjzz\") pod \"authorino-operator-75cc8d5cb7-895pz\" (UID: \"cc14b502-a312-4b50-a82d-3a9c3c158d91\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:15.795080 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.795048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjzz\" (UniqueName: \"kubernetes.io/projected/cc14b502-a312-4b50-a82d-3a9c3c158d91-kube-api-access-9bjzz\") pod \"authorino-operator-75cc8d5cb7-895pz\" (UID: \"cc14b502-a312-4b50-a82d-3a9c3c158d91\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:15.811736 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.811711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjzz\" (UniqueName: \"kubernetes.io/projected/cc14b502-a312-4b50-a82d-3a9c3c158d91-kube-api-access-9bjzz\") pod \"authorino-operator-75cc8d5cb7-895pz\" (UID: \"cc14b502-a312-4b50-a82d-3a9c3c158d91\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:15.910430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:15.910324 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:16.043478 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:16.043455 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-895pz"] Jul 29 06:34:16.045672 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:34:16.045644 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc14b502_a312_4b50_a82d_3a9c3c158d91.slice/crio-005d1bd3fe0f65d38b706297ec909701f19ae48cf6eb6e7d273cd76801c8dc34 WatchSource:0}: Error finding container 005d1bd3fe0f65d38b706297ec909701f19ae48cf6eb6e7d273cd76801c8dc34: Status 404 returned error can't find the container with id 005d1bd3fe0f65d38b706297ec909701f19ae48cf6eb6e7d273cd76801c8dc34 Jul 29 06:34:16.047746 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:16.047732 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 06:34:16.328612 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:16.328579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" event={"ID":"cc14b502-a312-4b50-a82d-3a9c3c158d91","Type":"ContainerStarted","Data":"005d1bd3fe0f65d38b706297ec909701f19ae48cf6eb6e7d273cd76801c8dc34"} Jul 29 06:34:16.947540 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:16.947499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:18.805246 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:18.805211 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs"] Jul 29 06:34:18.808581 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:18.808565 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:18.811444 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:18.811424 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-645m5\"" Jul 29 06:34:18.817495 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:18.817473 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs"] Jul 29 06:34:18.921220 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:18.921186 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfv7w\" (UniqueName: \"kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w\") pod \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" (UID: \"a64eb125-3e33-4573-84ae-3d6d780fddd9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:19.022036 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.022007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfv7w\" (UniqueName: \"kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w\") pod \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" (UID: \"a64eb125-3e33-4573-84ae-3d6d780fddd9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:19.033224 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.033195 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfv7w\" (UniqueName: \"kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w\") pod \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" (UID: \"a64eb125-3e33-4573-84ae-3d6d780fddd9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:19.118594 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.118517 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:19.180470 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.180441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:19.250792 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.250771 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs"] Jul 29 06:34:19.252921 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:34:19.252892 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64eb125_3e33_4573_84ae_3d6d780fddd9.slice/crio-e31fb25f00b3485129ddbb7128780d4ca66a312b6d688977b4db17d5978d0fae WatchSource:0}: Error finding container e31fb25f00b3485129ddbb7128780d4ca66a312b6d688977b4db17d5978d0fae: Status 404 returned error can't find the container with id e31fb25f00b3485129ddbb7128780d4ca66a312b6d688977b4db17d5978d0fae Jul 29 06:34:19.342141 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.342107 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" event={"ID":"a64eb125-3e33-4573-84ae-3d6d780fddd9","Type":"ContainerStarted","Data":"e31fb25f00b3485129ddbb7128780d4ca66a312b6d688977b4db17d5978d0fae"} Jul 29 06:34:19.343523 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.343500 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" event={"ID":"cc14b502-a312-4b50-a82d-3a9c3c158d91","Type":"ContainerStarted","Data":"caa89e902b60098c6c6b80e3e53c0b7dcb3503aa3d76151c126541f4844475f6"} Jul 29 06:34:19.343659 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.343627 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:19.361100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.361058 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" podStartSLOduration=2.054872246 podStartE2EDuration="4.361041532s" totalImagesPullingTime="2.306169286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:34:19.36011005 +0000 UTC m=+703.894189626" watchObservedRunningTime="2026-07-29 06:34:19.361041532 +0000 UTC m=+703.895121107" Jul 29 06:34:19.947683 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:19.947639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:20.966474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:20.966438 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:21.355499 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:21.355462 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" event={"ID":"a64eb125-3e33-4573-84ae-3d6d780fddd9","Type":"ContainerStarted","Data":"3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04"} Jul 29 06:34:21.355672 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:21.355535 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:21.370892 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:21.370716 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" podStartSLOduration=1.733383834 podStartE2EDuration="3.370700635s" totalImagesPullingTime="1.637316801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:34:21.370204488 +0000 UTC m=+705.904284062" watchObservedRunningTime="2026-07-29 06:34:21.370700635 +0000 UTC m=+705.904780205" Jul 29 06:34:22.996956 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:22.996918 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:23.595234 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:23.595190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:26.795810 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:26.795729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:28.751510 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:28.751469 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:28.854658 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:28.854625 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:29.146901 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:29.146816 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:29.247513 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:29.247472 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:30.351496 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:30.351469 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-895pz" Jul 29 06:34:32.361754 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:32.361725 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:32.555128 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:32.555091 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:32.655442 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:32.655343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:35.954567 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:35.954533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:36.054133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.054098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:36.264870 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.264837 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr"] Jul 29 06:34:36.269183 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.269166 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.271936 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.271915 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 29 06:34:36.271936 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.271932 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6llvq\"" Jul 29 06:34:36.272084 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.271946 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 29 06:34:36.275309 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.275290 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr"] Jul 29 06:34:36.283098 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.283073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:36.365656 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.365620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.365656 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.365660 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.365872 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.365692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-kube-api-access-48tmj\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.466487 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.466462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-kube-api-access-48tmj\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.466641 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.466528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.466641 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.466555 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.466712 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:36.466678 2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 29 06:34:36.466769 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:36.466757 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert podName:2d289693-1ca5-4e45-ba67-6ae353f4cf1f nodeName:}" failed. No retries permitted until 2026-07-29 06:34:36.966736077 +0000 UTC m=+721.500815646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-8w2tr" (UID: "2d289693-1ca5-4e45-ba67-6ae353f4cf1f") : secret "plugin-serving-cert" not found Jul 29 06:34:36.467152 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.467135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.477231 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.477208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-kube-api-access-48tmj\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.688014 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.687939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:36.971485 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.971456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:36.973888 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:36.973868 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d289693-1ca5-4e45-ba67-6ae353f4cf1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8w2tr\" (UID: \"2d289693-1ca5-4e45-ba67-6ae353f4cf1f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:37.178851 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:37.178815 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" Jul 29 06:34:37.315543 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:37.315515 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr"] Jul 29 06:34:37.318613 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:34:37.318585 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d289693_1ca5_4e45_ba67_6ae353f4cf1f.slice/crio-7bb44f73e3168d2ac5a30be1ce9853c5e48a9f092b6f2be135204ac6e4616c6d WatchSource:0}: Error finding container 7bb44f73e3168d2ac5a30be1ce9853c5e48a9f092b6f2be135204ac6e4616c6d: Status 404 returned error can't find the container with id 7bb44f73e3168d2ac5a30be1ce9853c5e48a9f092b6f2be135204ac6e4616c6d Jul 29 06:34:37.421306 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:37.421270 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" event={"ID":"2d289693-1ca5-4e45-ba67-6ae353f4cf1f","Type":"ContainerStarted","Data":"7bb44f73e3168d2ac5a30be1ce9853c5e48a9f092b6f2be135204ac6e4616c6d"} Jul 29 06:34:39.877333 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:39.877269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:39.978080 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:39.978040 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:46.559346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.559310 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs"] Jul 29 06:34:46.560192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.559578 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" containerName="manager" containerID="cri-o://3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04" gracePeriod=2 Jul 29 06:34:46.566839 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.566557 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs"] Jul 29 06:34:46.596236 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.595534 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr"] Jul 29 06:34:46.597088 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.596868 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" containerName="manager" Jul 29 06:34:46.597088 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.596894 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" containerName="manager" Jul 29 06:34:46.597088 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.596989 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" containerName="manager" Jul 29 06:34:46.601539 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.601515 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:46.604311 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:46.604281 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" is forbidden: User \"system:node:ip-10-0-131-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-31.ec2.internal' and this object" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:46.610561 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.610526 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr"] Jul 29 06:34:46.771945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.771907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mg8s\" (UniqueName: \"kubernetes.io/projected/0a85deae-4bc9-4d36-a487-bc132837791c-kube-api-access-6mg8s\") pod \"limitador-operator-controller-manager-859d7b78bd-s2lnr\" (UID: \"0a85deae-4bc9-4d36-a487-bc132837791c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:46.873325 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.873232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mg8s\" (UniqueName: \"kubernetes.io/projected/0a85deae-4bc9-4d36-a487-bc132837791c-kube-api-access-6mg8s\") pod \"limitador-operator-controller-manager-859d7b78bd-s2lnr\" (UID: \"0a85deae-4bc9-4d36-a487-bc132837791c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:46.885669 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.885640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mg8s\" (UniqueName: \"kubernetes.io/projected/0a85deae-4bc9-4d36-a487-bc132837791c-kube-api-access-6mg8s\") pod \"limitador-operator-controller-manager-859d7b78bd-s2lnr\" (UID: \"0a85deae-4bc9-4d36-a487-bc132837791c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:46.953405 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:46.953354 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:47.718297 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:47.718241 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:47.821695 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:47.821658 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:48.322090 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:48.322056 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:48.432027 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:48.431972 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:51.707778 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:51.707740 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:51.807361 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:51.807319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:54.111138 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:54.111101 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:54.211849 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:54.211794 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:58.182311 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.182285 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr"] Jul 29 06:34:58.184195 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:34:58.184169 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a85deae_4bc9_4d36_a487_bc132837791c.slice/crio-0e0297d192f1843dc9f94ab3b9c806b1d7a4b635b23a9c24a1abc7ecab4f3485 WatchSource:0}: Error finding container 0e0297d192f1843dc9f94ab3b9c806b1d7a4b635b23a9c24a1abc7ecab4f3485: Status 404 returned error can't find the container with id 0e0297d192f1843dc9f94ab3b9c806b1d7a4b635b23a9c24a1abc7ecab4f3485 Jul 29 06:34:58.191000 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.190983 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:58.193372 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:58.193346 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" is forbidden: User \"system:node:ip-10-0-131-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-31.ec2.internal' and this object" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:58.267658 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.267637 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfv7w\" (UniqueName: \"kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w\") pod \"a64eb125-3e33-4573-84ae-3d6d780fddd9\" (UID: \"a64eb125-3e33-4573-84ae-3d6d780fddd9\") " Jul 29 06:34:58.269547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.269524 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w" (OuterVolumeSpecName: "kube-api-access-dfv7w") pod "a64eb125-3e33-4573-84ae-3d6d780fddd9" (UID: "a64eb125-3e33-4573-84ae-3d6d780fddd9"). InnerVolumeSpecName "kube-api-access-dfv7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:34:58.369167 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.369092 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfv7w\" (UniqueName: \"kubernetes.io/projected/a64eb125-3e33-4573-84ae-3d6d780fddd9-kube-api-access-dfv7w\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:34:58.524654 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.524613 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" event={"ID":"2d289693-1ca5-4e45-ba67-6ae353f4cf1f","Type":"ContainerStarted","Data":"1c428336d83553aa70b3df2d4b17ff03fb52bce74804662011b7447100dbbc80"} Jul 29 06:34:58.525861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.525830 2556 generic.go:356] "Generic (PLEG): container finished" podID="a64eb125-3e33-4573-84ae-3d6d780fddd9" containerID="3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04" exitCode=0 Jul 29 06:34:58.525997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.525872 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:58.525997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.525922 2556 scope.go:117] "RemoveContainer" containerID="3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04" Jul 29 06:34:58.527231 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:58.527206 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" is forbidden: User \"system:node:ip-10-0-131-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-31.ec2.internal' and this object" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:58.527421 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.527400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" event={"ID":"0a85deae-4bc9-4d36-a487-bc132837791c","Type":"ContainerStarted","Data":"15e6b4e1a11a9cf331ec3d80c49219abc4577bc73263d5a16ed6ef99dd297732"} Jul 29 06:34:58.527487 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.527447 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" event={"ID":"0a85deae-4bc9-4d36-a487-bc132837791c","Type":"ContainerStarted","Data":"0e0297d192f1843dc9f94ab3b9c806b1d7a4b635b23a9c24a1abc7ecab4f3485"} Jul 29 06:34:58.527568 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.527548 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:34:58.535333 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.535311 2556 scope.go:117] "RemoveContainer" containerID="3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04" Jul 29 06:34:58.535602 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:58.535584 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04\": container with ID starting with 3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04 not found: ID does not exist" containerID="3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04" Jul 29 06:34:58.535659 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.535611 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04"} err="failed to get container status \"3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04\": rpc error: code = NotFound desc = could not find container \"3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04\": container with ID starting with 3f88027cb349d4788da878a4e962ebd1769b5e645a7f7fe2d20dabc4c76a6e04 not found: ID does not exist" Jul 29 06:34:58.543782 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.543742 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8w2tr" podStartSLOduration=1.7346522210000002 podStartE2EDuration="22.543730967s" totalImagesPullingTime="20.809078746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:34:58.541716414 +0000 UTC m=+743.075795987" watchObservedRunningTime="2026-07-29 06:34:58.543730967 +0000 UTC m=+743.077810539" Jul 29 06:34:58.544234 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:34:58.544205 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qf9qs\" is forbidden: User \"system:node:ip-10-0-131-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-31.ec2.internal' and this object" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qf9qs" Jul 29 06:34:58.566077 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.566027 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" podStartSLOduration=12.566009804 podStartE2EDuration="12.566009804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:34:58.563931095 +0000 UTC m=+743.098010669" watchObservedRunningTime="2026-07-29 06:34:58.566009804 +0000 UTC m=+743.100089377" Jul 29 06:34:58.662559 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.662488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:58.759868 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:58.759813 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:34:59.927606 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:34:59.927574 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64eb125-3e33-4573-84ae-3d6d780fddd9" path="/var/lib/kubelet/pods/a64eb125-3e33-4573-84ae-3d6d780fddd9/volumes" Jul 29 06:35:09.536446 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:09.536414 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s2lnr" Jul 29 06:35:15.266895 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.266858 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m"] Jul 29 06:35:15.269580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.269556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.272290 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.272270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4p5z6\"" Jul 29 06:35:15.279822 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.279799 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m"] Jul 29 06:35:15.292339 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.292315 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:15.296649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296627 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkhk\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-kube-api-access-trkhk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296728 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296728 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296729 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296748 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709b7a7-2b66-4056-909e-a24b27480c18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296772 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296888 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296819 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296920 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709b7a7-2b66-4056-909e-a24b27480c18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.296968 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.296943 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398200 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709b7a7-2b66-4056-909e-a24b27480c18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398241 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398303 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709b7a7-2b66-4056-909e-a24b27480c18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398330 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398367 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398362 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkhk\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-kube-api-access-trkhk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398717 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398900 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398762 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.398900 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.398811 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.399084 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.399055 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709b7a7-2b66-4056-909e-a24b27480c18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.401059 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.401036 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709b7a7-2b66-4056-909e-a24b27480c18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.401176 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.401160 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709b7a7-2b66-4056-909e-a24b27480c18-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.408834 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.408811 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.408984 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.408967 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkhk\" (UniqueName: \"kubernetes.io/projected/1709b7a7-2b66-4056-909e-a24b27480c18-kube-api-access-trkhk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-w7h2m\" (UID: \"1709b7a7-2b66-4056-909e-a24b27480c18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.580432 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.580353 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:15.628198 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.628147 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:15.711520 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.711495 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m"] Jul 29 06:35:15.712736 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:35:15.712710 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1709b7a7_2b66_4056_909e_a24b27480c18.slice/crio-be1a51933a9e65485c89e2faae60976560aed961fb520cfa177ba44012bd3abe WatchSource:0}: Error finding container be1a51933a9e65485c89e2faae60976560aed961fb520cfa177ba44012bd3abe: Status 404 returned error can't find the container with id be1a51933a9e65485c89e2faae60976560aed961fb520cfa177ba44012bd3abe Jul 29 06:35:15.714780 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.714752 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:35:15.714829 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.714817 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:35:15.714866 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:15.714855 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:35:16.606239 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:16.606200 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" event={"ID":"1709b7a7-2b66-4056-909e-a24b27480c18","Type":"ContainerStarted","Data":"74b6d630465b617ac7099a13a5859c084a784e73554fba5763be63088cd5f76f"} Jul 29 06:35:16.606662 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:16.606261 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" event={"ID":"1709b7a7-2b66-4056-909e-a24b27480c18","Type":"ContainerStarted","Data":"be1a51933a9e65485c89e2faae60976560aed961fb520cfa177ba44012bd3abe"} Jul 29 06:35:16.626101 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:16.626054 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" podStartSLOduration=1.626040461 podStartE2EDuration="1.626040461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:35:16.624705385 +0000 UTC m=+761.158784978" watchObservedRunningTime="2026-07-29 06:35:16.626040461 +0000 UTC m=+761.160120033" Jul 29 06:35:17.580923 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:17.580892 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:17.585619 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:17.585596 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:17.610280 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:17.610262 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:17.611186 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:17.611172 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:35:29.133039 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.132985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:29.191298 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.191257 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:29.241368 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.241333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:29.863967 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.863937 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:29.865934 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.865913 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:29.868414 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.868396 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-82x8j\"" Jul 29 06:35:29.872814 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.872789 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:29.891649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.891601 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:29.963358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:29.963330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:30.022483 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.022448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqx5\" (UniqueName: \"kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5\") pod \"authorino-58cd955675-zr48k\" (UID: \"795f9a60-346e-4de6-b184-5d06396d3380\") " pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:30.123945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.123869 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqx5\" (UniqueName: \"kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5\") pod \"authorino-58cd955675-zr48k\" (UID: \"795f9a60-346e-4de6-b184-5d06396d3380\") " pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:30.131777 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.131749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqx5\" (UniqueName: \"kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5\") pod \"authorino-58cd955675-zr48k\" (UID: \"795f9a60-346e-4de6-b184-5d06396d3380\") " pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:30.176099 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.176073 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:30.300930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.300878 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:30.303236 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:35:30.303209 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod795f9a60_346e_4de6_b184_5d06396d3380.slice/crio-ea5d6e7ee9056d6e7ede8c1cce8a270e5c5cc2449f155dc5638c60539eab5ad7 WatchSource:0}: Error finding container ea5d6e7ee9056d6e7ede8c1cce8a270e5c5cc2449f155dc5638c60539eab5ad7: Status 404 returned error can't find the container with id ea5d6e7ee9056d6e7ede8c1cce8a270e5c5cc2449f155dc5638c60539eab5ad7 Jul 29 06:35:30.672612 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:30.672573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zr48k" event={"ID":"795f9a60-346e-4de6-b184-5d06396d3380","Type":"ContainerStarted","Data":"ea5d6e7ee9056d6e7ede8c1cce8a270e5c5cc2449f155dc5638c60539eab5ad7"} Jul 29 06:35:33.688070 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:33.687989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zr48k" event={"ID":"795f9a60-346e-4de6-b184-5d06396d3380","Type":"ContainerStarted","Data":"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58"} Jul 29 06:35:33.697535 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:33.697513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:33.702358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:33.702320 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-zr48k" podStartSLOduration=1.658262604 podStartE2EDuration="4.702307192s" totalImagesPullingTime="3.044044588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:35:33.701321183 +0000 UTC m=+778.235400755" watchObservedRunningTime="2026-07-29 06:35:33.702307192 +0000 UTC m=+778.236386766" Jul 29 06:35:33.801319 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:33.801287 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:34.448974 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:34.448941 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:35.697022 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:35.696984 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-zr48k" podUID="795f9a60-346e-4de6-b184-5d06396d3380" containerName="authorino" containerID="cri-o://d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58" gracePeriod=30 Jul 29 06:35:35.944289 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:35.944265 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:36.080192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.080116 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bqx5\" (UniqueName: \"kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5\") pod \"795f9a60-346e-4de6-b184-5d06396d3380\" (UID: \"795f9a60-346e-4de6-b184-5d06396d3380\") " Jul 29 06:35:36.082295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.082263 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5" (OuterVolumeSpecName: "kube-api-access-4bqx5") pod "795f9a60-346e-4de6-b184-5d06396d3380" (UID: "795f9a60-346e-4de6-b184-5d06396d3380"). InnerVolumeSpecName "kube-api-access-4bqx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:35:36.181500 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.181472 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqx5\" (UniqueName: \"kubernetes.io/projected/795f9a60-346e-4de6-b184-5d06396d3380-kube-api-access-4bqx5\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:35:36.702251 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.702210 2556 generic.go:356] "Generic (PLEG): container finished" podID="795f9a60-346e-4de6-b184-5d06396d3380" containerID="d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58" exitCode=0 Jul 29 06:35:36.702700 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.702266 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zr48k" Jul 29 06:35:36.702700 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.702294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zr48k" event={"ID":"795f9a60-346e-4de6-b184-5d06396d3380","Type":"ContainerDied","Data":"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58"} Jul 29 06:35:36.702700 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.702336 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zr48k" event={"ID":"795f9a60-346e-4de6-b184-5d06396d3380","Type":"ContainerDied","Data":"ea5d6e7ee9056d6e7ede8c1cce8a270e5c5cc2449f155dc5638c60539eab5ad7"} Jul 29 06:35:36.702700 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.702353 2556 scope.go:117] "RemoveContainer" containerID="d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58" Jul 29 06:35:36.710942 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.710925 2556 scope.go:117] "RemoveContainer" containerID="d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58" Jul 29 06:35:36.711193 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:35:36.711175 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58\": container with ID starting with d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58 not found: ID does not exist" containerID="d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58" Jul 29 06:35:36.711248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.711202 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58"} err="failed to get container status \"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58\": rpc error: code = NotFound desc = could not find container \"d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58\": container with ID starting with d59bbcae20209d1c73e91c8938e3f029f4f25866c3b96a841e7c3974aa35cb58 not found: ID does not exist" Jul 29 06:35:36.725895 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.725875 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:36.728985 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:36.728963 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-zr48k"] Jul 29 06:35:37.927028 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:37.926996 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795f9a60-346e-4de6-b184-5d06396d3380" path="/var/lib/kubelet/pods/795f9a60-346e-4de6-b184-5d06396d3380/volumes" Jul 29 06:35:49.137671 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:49.137626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:49.238673 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:49.238639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:35:49.610894 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:35:49.610825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:01.842326 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:01.842236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:02.068464 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.068430 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:02.068784 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.068770 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f9a60-346e-4de6-b184-5d06396d3380" containerName="authorino" Jul 29 06:36:02.068828 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.068787 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f9a60-346e-4de6-b184-5d06396d3380" containerName="authorino" Jul 29 06:36:02.068861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.068840 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f9a60-346e-4de6-b184-5d06396d3380" containerName="authorino" Jul 29 06:36:02.071571 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.071554 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:02.074585 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.074565 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-82x8j\"" Jul 29 06:36:02.078887 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.078869 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:02.152227 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.152143 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:02.186333 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.186304 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7szx\" (UniqueName: \"kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx\") pod \"authorino-5d4b67c446-zhczb\" (UID: \"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede\") " pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:02.286920 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.286890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7szx\" (UniqueName: \"kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx\") pod \"authorino-5d4b67c446-zhczb\" (UID: \"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede\") " pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:02.294886 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.294859 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7szx\" (UniqueName: \"kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx\") pod \"authorino-5d4b67c446-zhczb\" (UID: \"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede\") " pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:02.303278 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.303255 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:02.303531 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.303520 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:02.373263 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.373232 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:36:02.376893 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.376871 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.379550 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.379533 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 06:36:02.385021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.384999 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:36:02.439080 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.439056 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:02.441331 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:36:02.441305 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b9b843f_6b9c_4ee0_b0ae_4802c1cdbede.slice/crio-4a1381d8d0f47f76844dbaa92a38787af371eb35e690d3b9ac64b4cfce913c3b WatchSource:0}: Error finding container 4a1381d8d0f47f76844dbaa92a38787af371eb35e690d3b9ac64b4cfce913c3b: Status 404 returned error can't find the container with id 4a1381d8d0f47f76844dbaa92a38787af371eb35e690d3b9ac64b4cfce913c3b Jul 29 06:36:02.488266 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.488227 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.488455 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.488272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5r6\" (UniqueName: \"kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.589708 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.589674 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.589856 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.589716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5r6\" (UniqueName: \"kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.592273 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.592254 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.597248 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.597228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5r6\" (UniqueName: \"kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6\") pod \"authorino-68cfdbfd88-pr4gg\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.688189 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.688160 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:36:02.814162 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.814142 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:36:02.816741 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:36:02.816714 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14456748_653b_4190_a093_e115b8d08e01.slice/crio-9fe40e6c850054bfd9a486dd146f6f1b9248c3c321f1ff34bc1c0c6046d1d462 WatchSource:0}: Error finding container 9fe40e6c850054bfd9a486dd146f6f1b9248c3c321f1ff34bc1c0c6046d1d462: Status 404 returned error can't find the container with id 9fe40e6c850054bfd9a486dd146f6f1b9248c3c321f1ff34bc1c0c6046d1d462 Jul 29 06:36:02.816962 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:02.816939 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zhczb" event={"ID":"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede","Type":"ContainerStarted","Data":"4a1381d8d0f47f76844dbaa92a38787af371eb35e690d3b9ac64b4cfce913c3b"} Jul 29 06:36:03.822371 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.822332 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zhczb" event={"ID":"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede","Type":"ContainerStarted","Data":"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581"} Jul 29 06:36:03.822870 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.822365 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-zhczb" podUID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" containerName="authorino" containerID="cri-o://ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581" gracePeriod=30 Jul 29 06:36:03.823885 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.823849 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" event={"ID":"14456748-653b-4190-a093-e115b8d08e01","Type":"ContainerStarted","Data":"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4"} Jul 29 06:36:03.823991 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.823894 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" event={"ID":"14456748-653b-4190-a093-e115b8d08e01","Type":"ContainerStarted","Data":"9fe40e6c850054bfd9a486dd146f6f1b9248c3c321f1ff34bc1c0c6046d1d462"} Jul 29 06:36:03.837805 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.837764 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-zhczb" podStartSLOduration=1.207007822 podStartE2EDuration="1.837748269s" totalImagesPullingTime="630.740447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:03.836052263 +0000 UTC m=+808.370131851" watchObservedRunningTime="2026-07-29 06:36:03.837748269 +0000 UTC m=+808.371827842" Jul 29 06:36:03.850063 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:03.850023 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" podStartSLOduration=1.40374789 podStartE2EDuration="1.850008736s" totalImagesPullingTime="446.260846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:03.849054052 +0000 UTC m=+808.383133625" watchObservedRunningTime="2026-07-29 06:36:03.850008736 +0000 UTC m=+808.384088309" Jul 29 06:36:04.068546 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.068520 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:04.104364 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.104305 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7szx\" (UniqueName: \"kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx\") pod \"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede\" (UID: \"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede\") " Jul 29 06:36:04.106390 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.106351 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx" (OuterVolumeSpecName: "kube-api-access-q7szx") pod "3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" (UID: "3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede"). InnerVolumeSpecName "kube-api-access-q7szx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:36:04.205680 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.205649 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7szx\" (UniqueName: \"kubernetes.io/projected/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede-kube-api-access-q7szx\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:36:04.828983 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.828948 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" containerID="ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581" exitCode=0 Jul 29 06:36:04.828983 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.828986 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zhczb" Jul 29 06:36:04.829556 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.829025 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zhczb" event={"ID":"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede","Type":"ContainerDied","Data":"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581"} Jul 29 06:36:04.829556 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.829067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zhczb" event={"ID":"3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede","Type":"ContainerDied","Data":"4a1381d8d0f47f76844dbaa92a38787af371eb35e690d3b9ac64b4cfce913c3b"} Jul 29 06:36:04.829556 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.829085 2556 scope.go:117] "RemoveContainer" containerID="ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581" Jul 29 06:36:04.837310 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.837293 2556 scope.go:117] "RemoveContainer" containerID="ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581" Jul 29 06:36:04.837586 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:36:04.837566 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581\": container with ID starting with ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581 not found: ID does not exist" containerID="ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581" Jul 29 06:36:04.837651 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.837594 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581"} err="failed to get container status \"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581\": rpc error: code = NotFound desc = could not find container \"ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581\": container with ID starting with ddc811a232056ff73bd4ac4c910b62f092e9e54a834057a83ceb9d3e1a53d581 not found: ID does not exist" Jul 29 06:36:04.848561 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.848541 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:04.852021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:04.852003 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zhczb"] Jul 29 06:36:05.926589 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:05.926558 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" path="/var/lib/kubelet/pods/3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede/volumes" Jul 29 06:36:08.542097 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:08.542061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:08.638847 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:08.638811 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:08.690388 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:08.690362 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:08.807840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:08.807745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:12.003493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:12.003453 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:12.102895 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:12.102860 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:43.603475 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:43.603439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:45.033100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:45.033064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:53.023801 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.023766 2556 generic.go:356] "Generic (PLEG): container finished" podID="1709b7a7-2b66-4056-909e-a24b27480c18" containerID="74b6d630465b617ac7099a13a5859c084a784e73554fba5763be63088cd5f76f" exitCode=0 Jul 29 06:36:53.024321 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.023851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" event={"ID":"1709b7a7-2b66-4056-909e-a24b27480c18","Type":"ContainerDied","Data":"74b6d630465b617ac7099a13a5859c084a784e73554fba5763be63088cd5f76f"} Jul 29 06:36:53.024321 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.024302 2556 scope.go:117] "RemoveContainer" containerID="74b6d630465b617ac7099a13a5859c084a784e73554fba5763be63088cd5f76f" Jul 29 06:36:53.025317 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.025292 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:36:53.025429 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.025359 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:36:53.025429 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:53.025408 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:36:54.030168 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.030132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" event={"ID":"1709b7a7-2b66-4056-909e-a24b27480c18","Type":"ContainerStarted","Data":"c9961837c0db0734b879655ddaea1179fd59bf77f529a9bdff6f3c47dda21579"} Jul 29 06:36:54.581321 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.581283 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:36:54.581524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.581489 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w7h2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Jul 29 06:36:54.581574 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.581544 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" podUID="1709b7a7-2b66-4056-909e-a24b27480c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Jul 29 06:36:54.794844 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.794801 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:54.895831 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:54.895741 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:36:55.581449 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:55.581410 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w7h2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Jul 29 06:36:55.581835 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:55.581470 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" podUID="1709b7a7-2b66-4056-909e-a24b27480c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Jul 29 06:36:56.581589 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:56.581553 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w7h2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Jul 29 06:36:56.581982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:56.581625 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" podUID="1709b7a7-2b66-4056-909e-a24b27480c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Jul 29 06:36:57.581424 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:57.581360 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-w7h2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Jul 29 06:36:57.581640 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:57.581444 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" podUID="1709b7a7-2b66-4056-909e-a24b27480c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Jul 29 06:36:58.584158 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:58.584129 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:36:58.584615 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:58.584404 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:36:58.585202 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:36:58.585188 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-w7h2m" Jul 29 06:37:01.427783 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:01.427748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:37:09.698119 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.698084 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx"] Jul 29 06:37:09.698607 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.698391 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" containerName="authorino" Jul 29 06:37:09.698607 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.698404 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" containerName="authorino" Jul 29 06:37:09.698607 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.698459 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9b843f-6b9c-4ee0-b0ae-4802c1cdbede" containerName="authorino" Jul 29 06:37:09.701630 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.701611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.704338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.704320 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 06:37:09.704557 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.704543 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kkwjc\"" Jul 29 06:37:09.705829 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.705812 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 06:37:09.705880 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.705811 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 06:37:09.714246 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.714223 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx"] Jul 29 06:37:09.768772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.768745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:37:09.849010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.848973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqj8\" (UniqueName: \"kubernetes.io/projected/98ddf9f5-19b7-4d93-b156-27db15d3db0b-kube-api-access-mjqj8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.849159 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.849019 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.849159 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.849055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.849159 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.849090 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.849287 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.849165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.849287 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.849193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.949797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.949797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqj8\" (UniqueName: \"kubernetes.io/projected/98ddf9f5-19b7-4d93-b156-27db15d3db0b-kube-api-access-mjqj8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.950003 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949811 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.950003 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949838 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.950003 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.950003 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.949896 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.950495 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.950361 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.951049 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.950497 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.951049 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.950441 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.952245 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.952215 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ddf9f5-19b7-4d93-b156-27db15d3db0b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.952567 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.952552 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ddf9f5-19b7-4d93-b156-27db15d3db0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:09.957894 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:09.957870 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqj8\" (UniqueName: \"kubernetes.io/projected/98ddf9f5-19b7-4d93-b156-27db15d3db0b-kube-api-access-mjqj8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t7jjx\" (UID: \"98ddf9f5-19b7-4d93-b156-27db15d3db0b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:10.011917 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:10.011881 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:10.143635 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:10.143610 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx"] Jul 29 06:37:10.146154 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:37:10.146127 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ddf9f5_19b7_4d93_b156_27db15d3db0b.slice/crio-32efdf0af4b5c8da45163051309823fefcd58da0d513c571d9da35aa20bc26f8 WatchSource:0}: Error finding container 32efdf0af4b5c8da45163051309823fefcd58da0d513c571d9da35aa20bc26f8: Status 404 returned error can't find the container with id 32efdf0af4b5c8da45163051309823fefcd58da0d513c571d9da35aa20bc26f8 Jul 29 06:37:11.102042 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:11.101996 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" event={"ID":"98ddf9f5-19b7-4d93-b156-27db15d3db0b","Type":"ContainerStarted","Data":"32efdf0af4b5c8da45163051309823fefcd58da0d513c571d9da35aa20bc26f8"} Jul 29 06:37:12.107981 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:12.107945 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" event={"ID":"98ddf9f5-19b7-4d93-b156-27db15d3db0b","Type":"ContainerStarted","Data":"a1818c4c3f4135eda67ed71266cdc9e8ad6da9dabd46452b38801e2640873cb9"} Jul 29 06:37:12.125331 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:12.125262 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" podStartSLOduration=2.084000289 podStartE2EDuration="3.125243994s" totalImagesPullingTime="1.041243705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:37:12.124607645 +0000 UTC m=+876.658687218" watchObservedRunningTime="2026-07-29 06:37:12.125243994 +0000 UTC m=+876.659323569" Jul 29 06:37:15.019869 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:15.019827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:37:20.012069 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:20.012032 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:20.012569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:20.012102 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:20.024171 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:20.024149 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:20.151506 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:20.151475 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t7jjx" Jul 29 06:37:29.896305 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:29.896203 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:37:56.610753 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.610719 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst"] Jul 29 06:37:56.614168 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.614149 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.616939 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.616920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 29 06:37:56.623455 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.623431 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst"] Jul 29 06:37:56.638210 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.638187 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:37:56.746759 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65fa25-7948-490a-9003-ab78da495b65-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.746931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746768 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2jf\" (UniqueName: \"kubernetes.io/projected/ef65fa25-7948-490a-9003-ab78da495b65-kube-api-access-zf2jf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.746931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.746931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.746931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.746931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.746931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848239 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848292 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65fa25-7948-490a-9003-ab78da495b65-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848316 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2jf\" (UniqueName: \"kubernetes.io/projected/ef65fa25-7948-490a-9003-ab78da495b65-kube-api-access-zf2jf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848340 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848719 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848780 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848705 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.848780 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.848771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.851021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.850989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65fa25-7948-490a-9003-ab78da495b65-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.851179 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.851160 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65fa25-7948-490a-9003-ab78da495b65-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.856356 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.856335 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2jf\" (UniqueName: \"kubernetes.io/projected/ef65fa25-7948-490a-9003-ab78da495b65-kube-api-access-zf2jf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ghfst\" (UID: \"ef65fa25-7948-490a-9003-ab78da495b65\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:56.926665 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:56.926569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:37:57.052793 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:57.052767 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst"] Jul 29 06:37:57.055143 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:37:57.055115 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef65fa25_7948_490a_9003_ab78da495b65.slice/crio-a8bb3e67df6ef43f812c8b55defec59f68427233ceade0e82e2de6197aadf699 WatchSource:0}: Error finding container a8bb3e67df6ef43f812c8b55defec59f68427233ceade0e82e2de6197aadf699: Status 404 returned error can't find the container with id a8bb3e67df6ef43f812c8b55defec59f68427233ceade0e82e2de6197aadf699 Jul 29 06:37:57.299910 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:57.299881 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" event={"ID":"ef65fa25-7948-490a-9003-ab78da495b65","Type":"ContainerStarted","Data":"a8bb3e67df6ef43f812c8b55defec59f68427233ceade0e82e2de6197aadf699"} Jul 29 06:37:58.305753 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:58.305710 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" event={"ID":"ef65fa25-7948-490a-9003-ab78da495b65","Type":"ContainerStarted","Data":"24cd190c584349c28764c93423869a3198daa78b19d3624c2c48b7e773ca1a67"} Jul 29 06:37:58.322511 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:58.322468 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" podStartSLOduration=2.138569961 podStartE2EDuration="2.322453097s" totalImagesPullingTime="183.883136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:37:58.321223546 +0000 UTC m=+922.855303119" watchObservedRunningTime="2026-07-29 06:37:58.322453097 +0000 UTC m=+922.856532674" Jul 29 06:37:58.393228 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:37:58.393194 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:38:04.286829 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.286802 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg"] Jul 29 06:38:04.290253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.290228 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.292875 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.292854 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 29 06:38:04.300714 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.300693 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg"] Jul 29 06:38:04.312091 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.312053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:38:04.411918 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.411880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.412081 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.411928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b89e206-30a3-4462-b629-262e6dfba4e9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.412081 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.411949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.412081 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.412056 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdm5\" (UniqueName: \"kubernetes.io/projected/6b89e206-30a3-4462-b629-262e6dfba4e9-kube-api-access-gtdm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.412231 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.412094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.412231 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.412117 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513363 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513331 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdm5\" (UniqueName: \"kubernetes.io/projected/6b89e206-30a3-4462-b629-262e6dfba4e9-kube-api-access-gtdm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513577 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513370 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513577 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513577 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513473 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513743 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b89e206-30a3-4462-b629-262e6dfba4e9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513743 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513660 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513844 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.513928 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.513910 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.514056 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.514035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.515983 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.515964 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b89e206-30a3-4462-b629-262e6dfba4e9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.516254 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.516234 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b89e206-30a3-4462-b629-262e6dfba4e9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.520967 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.520945 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdm5\" (UniqueName: \"kubernetes.io/projected/6b89e206-30a3-4462-b629-262e6dfba4e9-kube-api-access-gtdm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg\" (UID: \"6b89e206-30a3-4462-b629-262e6dfba4e9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.600632 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.600556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:04.726356 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:04.726333 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg"] Jul 29 06:38:04.728270 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:38:04.728239 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b89e206_30a3_4462_b629_262e6dfba4e9.slice/crio-3b918c93da93f088805e593f0ea4c8522c4950a5bb28cd0af05c629a9bc9fd25 WatchSource:0}: Error finding container 3b918c93da93f088805e593f0ea4c8522c4950a5bb28cd0af05c629a9bc9fd25: Status 404 returned error can't find the container with id 3b918c93da93f088805e593f0ea4c8522c4950a5bb28cd0af05c629a9bc9fd25 Jul 29 06:38:05.334982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:05.334948 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" event={"ID":"6b89e206-30a3-4462-b629-262e6dfba4e9","Type":"ContainerStarted","Data":"6837fdab052e5408269f816e56f597aa688ea4e7aaecdfaaae1a5c6075f17bc0"} Jul 29 06:38:05.334982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:05.334985 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" event={"ID":"6b89e206-30a3-4462-b629-262e6dfba4e9","Type":"ContainerStarted","Data":"3b918c93da93f088805e593f0ea4c8522c4950a5bb28cd0af05c629a9bc9fd25"} Jul 29 06:38:05.352473 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:05.352428 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" podStartSLOduration=1.170414538 podStartE2EDuration="1.352410064s" totalImagesPullingTime="181.995526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:38:05.350731969 +0000 UTC m=+929.884811559" watchObservedRunningTime="2026-07-29 06:38:05.352410064 +0000 UTC m=+929.886489637" Jul 29 06:38:06.927193 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:06.927149 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:38:06.927611 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:06.927232 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:38:06.939982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:06.939961 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:38:07.355982 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:07.355953 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ghfst" Jul 29 06:38:14.600793 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:14.600757 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:14.600793 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:14.600795 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:14.613494 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:14.613463 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:15.391396 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:15.391354 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg" Jul 29 06:38:39.277134 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:39.277081 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:38:41.781545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:38:41.781505 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:39:04.481539 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.481449 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:39:04.482017 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.481744 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" podUID="14456748-653b-4190-a093-e115b8d08e01" containerName="authorino" containerID="cri-o://1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4" gracePeriod=30 Jul 29 06:39:04.731037 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.731015 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:39:04.843588 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.843510 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert\") pod \"14456748-653b-4190-a093-e115b8d08e01\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " Jul 29 06:39:04.843588 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.843575 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb5r6\" (UniqueName: \"kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6\") pod \"14456748-653b-4190-a093-e115b8d08e01\" (UID: \"14456748-653b-4190-a093-e115b8d08e01\") " Jul 29 06:39:04.846025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.845997 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6" (OuterVolumeSpecName: "kube-api-access-zb5r6") pod "14456748-653b-4190-a093-e115b8d08e01" (UID: "14456748-653b-4190-a093-e115b8d08e01"). InnerVolumeSpecName "kube-api-access-zb5r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:39:04.853696 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.853667 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "14456748-653b-4190-a093-e115b8d08e01" (UID: "14456748-653b-4190-a093-e115b8d08e01"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 06:39:04.944418 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.944360 2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14456748-653b-4190-a093-e115b8d08e01-tls-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:39:04.944418 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.944413 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5r6\" (UniqueName: \"kubernetes.io/projected/14456748-653b-4190-a093-e115b8d08e01-kube-api-access-zb5r6\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:39:04.979950 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:04.979911 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:39:05.080557 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.080532 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:39:05.599154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.599113 2556 generic.go:356] "Generic (PLEG): container finished" podID="14456748-653b-4190-a093-e115b8d08e01" containerID="1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4" exitCode=0 Jul 29 06:39:05.599154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.599154 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" Jul 29 06:39:05.599650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.599163 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" event={"ID":"14456748-653b-4190-a093-e115b8d08e01","Type":"ContainerDied","Data":"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4"} Jul 29 06:39:05.599650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.599208 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cfdbfd88-pr4gg" event={"ID":"14456748-653b-4190-a093-e115b8d08e01","Type":"ContainerDied","Data":"9fe40e6c850054bfd9a486dd146f6f1b9248c3c321f1ff34bc1c0c6046d1d462"} Jul 29 06:39:05.599650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.599241 2556 scope.go:117] "RemoveContainer" containerID="1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4" Jul 29 06:39:05.608613 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.608598 2556 scope.go:117] "RemoveContainer" containerID="1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4" Jul 29 06:39:05.608878 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:39:05.608860 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4\": container with ID starting with 1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4 not found: ID does not exist" containerID="1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4" Jul 29 06:39:05.608939 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.608888 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4"} err="failed to get container status \"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4\": rpc error: code = NotFound desc = could not find container \"1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4\": container with ID starting with 1239c260bbce2a94abd2b83a7ad12c57897c98ae8e5274596c829ddd2bf271d4 not found: ID does not exist" Jul 29 06:39:05.626873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.626833 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:39:05.641483 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.641458 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68cfdbfd88-pr4gg"] Jul 29 06:39:05.927283 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:05.927210 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14456748-653b-4190-a093-e115b8d08e01" path="/var/lib/kubelet/pods/14456748-653b-4190-a093-e115b8d08e01/volumes" Jul 29 06:39:06.122821 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:06.122786 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:39:06.223745 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:06.223713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:39:51.188050 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:39:51.188009 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:40:27.653123 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:40:27.653040 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:10.504005 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:10.503968 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:11.767687 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:11.767651 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:35.247692 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:35.247647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:35.348661 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:35.348619 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:43.783041 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:43.783004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:43.882130 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:43.882092 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:44.004077 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:44.004041 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:49.673110 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:49.673047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:50.787514 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:50.787480 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:55.495400 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:55.495332 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:41:55.560085 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:41:55.560021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:45:29.303935 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:45:29.303897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:46:34.559518 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:46:34.559433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:47:02.207597 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:47:02.207559 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:47:28.469175 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:47:28.469141 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:49:51.610496 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.610418 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:49:51.610980 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.610756 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14456748-653b-4190-a093-e115b8d08e01" containerName="authorino" Jul 29 06:49:51.610980 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.610768 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14456748-653b-4190-a093-e115b8d08e01" containerName="authorino" Jul 29 06:49:51.610980 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.610824 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14456748-653b-4190-a093-e115b8d08e01" containerName="authorino" Jul 29 06:49:51.613560 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.613545 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.616208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.616189 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-697d22b0-kserve-self-signed-certs\"" Jul 29 06:49:51.626881 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.626857 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:49:51.635480 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635458 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrwh\" (UniqueName: \"kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.635581 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.635625 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.635659 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635618 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.635659 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.635725 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.635661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.657728 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.657700 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:49:51.736422 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736575 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736468 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736575 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736575 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736575 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736552 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736776 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrwh\" (UniqueName: \"kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736898 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736874 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736898 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736891 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.736986 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.736957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.738772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.738750 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.739058 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.739041 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.744288 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.744266 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrwh\" (UniqueName: \"kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh\") pod \"e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:51.922078 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:51.921998 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:49:52.051518 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:52.051493 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:49:52.053184 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:49:52.053159 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ac73e2_e993_4e91_bc74_04d0e18543d6.slice/crio-e3b70eff354e12c0ce37541b36ed8253d3b4afe89f8c23ae617134d52d284733 WatchSource:0}: Error finding container e3b70eff354e12c0ce37541b36ed8253d3b4afe89f8c23ae617134d52d284733: Status 404 returned error can't find the container with id e3b70eff354e12c0ce37541b36ed8253d3b4afe89f8c23ae617134d52d284733 Jul 29 06:49:52.315260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:52.315228 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" event={"ID":"a5ac73e2-e993-4e91-bc74-04d0e18543d6","Type":"ContainerStarted","Data":"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14"} Jul 29 06:49:52.315260 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:52.315269 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" event={"ID":"a5ac73e2-e993-4e91-bc74-04d0e18543d6","Type":"ContainerStarted","Data":"e3b70eff354e12c0ce37541b36ed8253d3b4afe89f8c23ae617134d52d284733"} Jul 29 06:49:52.333454 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:49:52.333413 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" podStartSLOduration=1.333397963 podStartE2EDuration="1.333397963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:49:52.331480075 +0000 UTC m=+1636.865559661" watchObservedRunningTime="2026-07-29 06:49:52.333397963 +0000 UTC m=+1636.867477534" Jul 29 06:50:00.697183 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:00.697148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:50:01.927810 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:01.927719 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:01.927810 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:01.927758 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:01.935706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:01.935682 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:02.381023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:02.380998 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:18.984555 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:18.984522 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:50:19.004197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:19.004164 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:50:19.861165 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:19.861110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:50:34.003729 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.003688 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" podUID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" containerName="main" containerID="cri-o://d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14" gracePeriod=45 Jul 29 06:50:34.251727 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.251704 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:34.306638 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306567 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.306638 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306613 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.306840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306648 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.306840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306713 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.306840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306741 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.306840 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306822 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svrwh\" (UniqueName: \"kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh\") pod \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\" (UID: \"a5ac73e2-e993-4e91-bc74-04d0e18543d6\") " Jul 29 06:50:34.307037 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.306944 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:50:34.307141 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.307103 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home" (OuterVolumeSpecName: "home") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:50:34.307222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.307147 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:50:34.307222 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.307164 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.309237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.309211 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm" (OuterVolumeSpecName: "dshm") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:50:34.309237 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.309214 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 06:50:34.309548 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.309524 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh" (OuterVolumeSpecName: "kube-api-access-svrwh") pod "a5ac73e2-e993-4e91-bc74-04d0e18543d6" (UID: "a5ac73e2-e993-4e91-bc74-04d0e18543d6"). InnerVolumeSpecName "kube-api-access-svrwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:50:34.408336 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.408305 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.408336 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.408330 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.408336 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.408339 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrwh\" (UniqueName: \"kubernetes.io/projected/a5ac73e2-e993-4e91-bc74-04d0e18543d6-kube-api-access-svrwh\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.408569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.408350 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ac73e2-e993-4e91-bc74-04d0e18543d6-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.408569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.408358 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ac73e2-e993-4e91-bc74-04d0e18543d6-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:50:34.505344 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.505309 2556 generic.go:356] "Generic (PLEG): container finished" podID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" containerID="d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14" exitCode=0 Jul 29 06:50:34.505524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.505354 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" Jul 29 06:50:34.505524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.505353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" event={"ID":"a5ac73e2-e993-4e91-bc74-04d0e18543d6","Type":"ContainerDied","Data":"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14"} Jul 29 06:50:34.505524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.505403 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n" event={"ID":"a5ac73e2-e993-4e91-bc74-04d0e18543d6","Type":"ContainerDied","Data":"e3b70eff354e12c0ce37541b36ed8253d3b4afe89f8c23ae617134d52d284733"} Jul 29 06:50:34.505524 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.505424 2556 scope.go:117] "RemoveContainer" containerID="d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14" Jul 29 06:50:34.514633 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.514617 2556 scope.go:117] "RemoveContainer" containerID="d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14" Jul 29 06:50:34.514884 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:50:34.514867 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14\": container with ID starting with d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14 not found: ID does not exist" containerID="d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14" Jul 29 06:50:34.514930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.514892 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14"} err="failed to get container status \"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14\": rpc error: code = NotFound desc = could not find container \"d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14\": container with ID starting with d0c9ac78695ea95fd4255505fc79287a692ea1837e7654b750b9e995f5831c14 not found: ID does not exist" Jul 29 06:50:34.525945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.525921 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:50:34.531140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:34.531120 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-697d22b0-kserve-5f5fdb95f-xbg9n"] Jul 29 06:50:35.927514 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:50:35.927477 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" path="/var/lib/kubelet/pods/a5ac73e2-e993-4e91-bc74-04d0e18543d6/volumes" Jul 29 06:53:27.436219 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:27.436186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:47.263798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:47.263755 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:50.495196 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:50.495159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:50.593250 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:50.593218 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:50.989684 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:50.989655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:56.607121 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:56.607039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:56.666151 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:56.666116 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:56.820902 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:56.820845 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:57.102865 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.102833 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:53:57.103168 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.103155 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" containerName="main" Jul 29 06:53:57.103214 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.103170 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" containerName="main" Jul 29 06:53:57.103261 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.103250 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ac73e2-e993-4e91-bc74-04d0e18543d6" containerName="main" Jul 29 06:53:57.106242 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.106220 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.109021 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.109000 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-93562a48-openshift-default-dockercfg-tr96p\"" Jul 29 06:53:57.126495 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.126466 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:53:57.130294 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.130265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:57.214426 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214457 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214484 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214608 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xxg\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214639 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214658 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.214880 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.214761 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.215685 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.215658 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:57.316003 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.315971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xxg\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316197 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316236 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316311 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316452 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316340 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316652 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316578 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316656 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316706 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316687 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.316966 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.316753 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.317049 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.317031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.318443 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.318419 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.318732 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.318717 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.323604 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.323586 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.324013 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.323992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xxg\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg\") pod \"e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.416448 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.416357 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:57.552292 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.552261 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:53:57.555486 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:53:57.555451 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2627d8e_caf6_4946_8b77_3ef390ae2e9c.slice/crio-d5379e4b25cd4905c49c8860724065695d29e94de20ea389bf47c9895338de5a WatchSource:0}: Error finding container d5379e4b25cd4905c49c8860724065695d29e94de20ea389bf47c9895338de5a: Status 404 returned error can't find the container with id d5379e4b25cd4905c49c8860724065695d29e94de20ea389bf47c9895338de5a Jul 29 06:53:57.557975 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.557942 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:53:57.558065 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.558016 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:53:57.558125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:57.558061 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:53:58.366249 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:58.366216 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerStarted","Data":"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467"} Jul 29 06:53:58.366709 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:58.366268 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerStarted","Data":"d5379e4b25cd4905c49c8860724065695d29e94de20ea389bf47c9895338de5a"} Jul 29 06:53:58.387068 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:58.387025 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podStartSLOduration=1.387010622 podStartE2EDuration="1.387010622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:53:58.385781913 +0000 UTC m=+1882.919861521" watchObservedRunningTime="2026-07-29 06:53:58.387010622 +0000 UTC m=+1882.921090326" Jul 29 06:53:58.417513 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:58.417486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:58.421710 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:58.421688 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:59.370728 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:59.370686 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:59.371580 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:59.371561 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:53:59.721438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:59.721396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:59.798630 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:59.798603 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:53:59.959048 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:53:59.959016 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:00.185806 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:00.185729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:00.359038 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:00.358992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:00.456352 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:00.456317 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:11.645821 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:11.645785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:11.693903 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:11.693867 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:14.841055 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:14.841018 2556 patch_prober.go:28] interesting pod/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": read tcp 10.132.0.2:54138->10.132.0.31:15021: read: connection reset by peer" start-of-body= Jul 29 06:54:14.841488 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:14.841107 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": read tcp 10.132.0.2:54138->10.132.0.31:15021: read: connection reset by peer" Jul 29 06:54:15.439128 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.439096 2556 generic.go:356] "Generic (PLEG): container finished" podID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerID="e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467" exitCode=0 Jul 29 06:54:15.439290 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.439159 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerDied","Data":"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467"} Jul 29 06:54:15.439560 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.439545 2556 scope.go:117] "RemoveContainer" containerID="e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467" Jul 29 06:54:15.440543 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.440516 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:15.440609 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.440587 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:15.440646 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:15.440630 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:16.445126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:16.445093 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerStarted","Data":"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f"} Jul 29 06:54:17.417048 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:17.417004 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:17.417344 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:17.417317 2556 patch_prober.go:28] interesting pod/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 06:54:17.417484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:17.417418 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 06:54:18.416984 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:18.416947 2556 patch_prober.go:28] interesting pod/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 06:54:18.417369 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:18.417017 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 06:54:19.416824 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:19.416793 2556 patch_prober.go:28] interesting pod/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 06:54:19.416988 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:19.416854 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 06:54:20.420757 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:20.420724 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:20.421117 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:20.420981 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:20.421754 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:20.421739 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:40.464029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.463985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:40.531427 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.531396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:40.597199 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.597173 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:40.652445 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.652409 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:40.775890 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.775796 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:54:40.776095 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.776070 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" containerID="cri-o://b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f" gracePeriod=30 Jul 29 06:54:40.800035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:40.800000 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.026653 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.026567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.301198 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.301114 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.398164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.398130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.497150 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.497114 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.838217 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.838189 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:54:41.840253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.840237 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.846015 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.845993 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-89063c4d-gw-openshift-default-dockercfg-g9225\"" Jul 29 06:54:41.865104 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.865083 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:54:41.867676 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.867654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:41.903848 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.903806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.903866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.903899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.903926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.903965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904240 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.904052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904240 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.904078 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904240 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.904102 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:41.904429 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:41.904260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnkzr\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005741 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005705 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnkzr\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005748 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005788 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005807 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.005930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.005956 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006253 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.006206 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.006307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.006363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.006396 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.006749 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.006592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.008238 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.008218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.008586 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.008565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.016999 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.016947 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.019125 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.019105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnkzr\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr\") pod \"e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.149988 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.149892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:42.279514 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.279485 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:54:42.281985 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:54:42.281954 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de3ea20_896e_4605_b13b_302b03e46e75.slice/crio-f9296c673a13fee16f93a7b6c97951cf099b086ed34f7487f36fefd0ccf7c59d WatchSource:0}: Error finding container f9296c673a13fee16f93a7b6c97951cf099b086ed34f7487f36fefd0ccf7c59d: Status 404 returned error can't find the container with id f9296c673a13fee16f93a7b6c97951cf099b086ed34f7487f36fefd0ccf7c59d Jul 29 06:54:42.284100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.284071 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:42.284166 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.284153 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:42.284202 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.284182 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:42.467221 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.467189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:42.563221 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.563192 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerStarted","Data":"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b"} Jul 29 06:54:42.563708 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.563252 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerStarted","Data":"f9296c673a13fee16f93a7b6c97951cf099b086ed34f7487f36fefd0ccf7c59d"} Jul 29 06:54:42.583136 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:42.583084 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podStartSLOduration=1.583069303 podStartE2EDuration="1.583069303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:54:42.580504276 +0000 UTC m=+1927.114583893" watchObservedRunningTime="2026-07-29 06:54:42.583069303 +0000 UTC m=+1927.117148957" Jul 29 06:54:43.150464 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:43.150432 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:43.154783 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:43.154759 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:43.567252 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:43.567226 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:43.568110 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:43.568089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:54:44.040048 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:44.040017 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:44.151210 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:44.151183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:44.251731 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:44.251695 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:44.639372 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:44.639330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:44.744461 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:44.744430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:54:46.021693 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.021667 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:46.042772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042675 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.042772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042733 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042777 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042846 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042870 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9xxg\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042891 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042944 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042948 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042966 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.042992 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data\") pod \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\" (UID: \"a2627d8e-caf6-4946-8b77-3ef390ae2e9c\") " Jul 29 06:54:46.043425 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043229 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:54:46.043425 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043301 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 06:54:46.043545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043440 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.043545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043458 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.043545 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043475 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.043695 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.043634 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:54:46.044543 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.044514 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:54:46.046431 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.046405 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg" (OuterVolumeSpecName: "kube-api-access-k9xxg") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "kube-api-access-k9xxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:54:46.046532 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.046418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 06:54:46.046532 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.046477 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:54:46.046639 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.046591 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2627d8e-caf6-4946-8b77-3ef390ae2e9c" (UID: "a2627d8e-caf6-4946-8b77-3ef390ae2e9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:54:46.144127 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144097 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.144127 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144127 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.144307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144138 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.144307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144150 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.144307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144161 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xxg\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-kube-api-access-k9xxg\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.144307 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.144172 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2627d8e-caf6-4946-8b77-3ef390ae2e9c-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:54:46.587594 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.587554 2556 generic.go:356] "Generic (PLEG): container finished" podID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerID="b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f" exitCode=0 Jul 29 06:54:46.587750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.587606 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" Jul 29 06:54:46.587750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.587616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerDied","Data":"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f"} Jul 29 06:54:46.587750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.587672 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp" event={"ID":"a2627d8e-caf6-4946-8b77-3ef390ae2e9c","Type":"ContainerDied","Data":"d5379e4b25cd4905c49c8860724065695d29e94de20ea389bf47c9895338de5a"} Jul 29 06:54:46.587750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.587704 2556 scope.go:117] "RemoveContainer" containerID="b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f" Jul 29 06:54:46.597769 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.597749 2556 scope.go:117] "RemoveContainer" containerID="e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467" Jul 29 06:54:46.605119 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.605099 2556 scope.go:117] "RemoveContainer" containerID="b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f" Jul 29 06:54:46.605357 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:54:46.605337 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f\": container with ID starting with b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f not found: ID does not exist" containerID="b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f" Jul 29 06:54:46.605422 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.605364 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f"} err="failed to get container status \"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f\": rpc error: code = NotFound desc = could not find container \"b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f\": container with ID starting with b51a03f31d87153dc735660da18620526de5bfd47834c6916529317ee730c16f not found: ID does not exist" Jul 29 06:54:46.605422 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.605398 2556 scope.go:117] "RemoveContainer" containerID="e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467" Jul 29 06:54:46.605612 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:54:46.605594 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467\": container with ID starting with e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467 not found: ID does not exist" containerID="e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467" Jul 29 06:54:46.605654 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.605618 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467"} err="failed to get container status \"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467\": rpc error: code = NotFound desc = could not find container \"e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467\": container with ID starting with e107eb1bcf5611379f6fa9e1a462dbf931616f23ea8a3c0539bcc8ade60ad467 not found: ID does not exist" Jul 29 06:54:46.609704 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.609682 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:54:46.612740 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:46.612720 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-93562a48-openshift-default-86f77b96dd-25nqp"] Jul 29 06:54:47.927650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:47.927614 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" path="/var/lib/kubelet/pods/a2627d8e-caf6-4946-8b77-3ef390ae2e9c/volumes" Jul 29 06:54:58.631058 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:58.630197 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:38760->10.132.0.32:15021: read: connection reset by peer" start-of-body= Jul 29 06:54:58.631058 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:58.630295 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:38760->10.132.0.32:15021: read: connection reset by peer" Jul 29 06:54:59.647637 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.647601 2556 generic.go:356] "Generic (PLEG): container finished" podID="3de3ea20-896e-4605-b13b-302b03e46e75" containerID="de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b" exitCode=0 Jul 29 06:54:59.648060 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.647672 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerDied","Data":"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b"} Jul 29 06:54:59.648060 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.648027 2556 scope.go:117] "RemoveContainer" containerID="de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b" Jul 29 06:54:59.648983 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.648954 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:59.649103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.649038 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:54:59.649103 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:54:59.649084 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:00.656986 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:00.656946 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerStarted","Data":"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e"} Jul 29 06:55:01.150296 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:01.150251 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:01.150532 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:01.150510 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 29 06:55:01.150613 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:01.150568 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 29 06:55:02.151231 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:02.151191 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 29 06:55:02.152730 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:02.152697 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 29 06:55:03.150291 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:03.150256 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 29 06:55:03.150783 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:03.150750 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 29 06:55:04.150881 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:04.150846 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 29 06:55:04.151451 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:04.151416 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 29 06:55:05.154358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:05.154327 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:05.154734 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:05.154716 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:05.155387 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:05.155358 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:14.648641 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.648605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:14.716964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.716931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:14.770971 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.770939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:14.852564 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.852508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:14.985670 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.985636 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:55:14.985910 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:14.985887 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" containerID="cri-o://853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e" gracePeriod=30 Jul 29 06:55:15.012108 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.012080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:15.225950 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.225916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:15.469081 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.469044 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:15.757978 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.757904 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:15.758426 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758406 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.758493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758429 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.758493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758449 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.758493 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758458 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.758650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758543 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.758650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.758559 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2627d8e-caf6-4946-8b77-3ef390ae2e9c" containerName="istio-proxy" Jul 29 06:55:15.761336 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.761313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.764044 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.764025 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-88aa9015-openshift-default-dockercfg-bmv9n\"" Jul 29 06:55:15.774009 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.773987 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:15.796717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.796674 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:15.912744 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912712 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912744 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912773 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912805 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912843 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.912997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.912971 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:15.913334 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:15.913007 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7kp\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7kp\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014175 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014273 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014298 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014342 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014435 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.014861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014850 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.015087 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014903 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.015087 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.014957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.015259 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.015240 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.016816 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.016797 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.017025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.017008 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.023029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.023004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7kp\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.023175 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.023159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token\") pod \"e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.071322 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.071298 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:16.208778 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.208667 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:16.211063 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:55:16.211023 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0732e37_fced_4541_ba28_731a2a57e6a9.slice/crio-5a1aa4adfadbfc116695a1b4129e0355954a251ee19acbc5713aec4fe3028a72 WatchSource:0}: Error finding container 5a1aa4adfadbfc116695a1b4129e0355954a251ee19acbc5713aec4fe3028a72: Status 404 returned error can't find the container with id 5a1aa4adfadbfc116695a1b4129e0355954a251ee19acbc5713aec4fe3028a72 Jul 29 06:55:16.213154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.213127 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:16.213211 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.213192 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:16.213254 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.213219 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:16.280164 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.280105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:16.735485 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.735449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerStarted","Data":"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65"} Jul 29 06:55:16.735665 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.735507 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerStarted","Data":"5a1aa4adfadbfc116695a1b4129e0355954a251ee19acbc5713aec4fe3028a72"} Jul 29 06:55:16.755917 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:16.755870 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podStartSLOduration=1.755853799 podStartE2EDuration="1.755853799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:55:16.754343962 +0000 UTC m=+1961.288423535" watchObservedRunningTime="2026-07-29 06:55:16.755853799 +0000 UTC m=+1961.289933442" Jul 29 06:55:17.072582 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:17.072491 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:17.077547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:17.077522 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:17.739634 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:17.739603 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:17.740593 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:17.740569 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:18.094163 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.094088 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:18.181405 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.181364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:18.242346 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.242319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:18.539589 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.539557 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:18.819860 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.819772 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:18.904803 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:18.904748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:20.246129 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.246071 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:20.349534 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349505 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnkzr\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.349720 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349548 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.349720 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349592 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.349720 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349613 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.349720 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349676 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.349720 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349693 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.350008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349757 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.350008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349779 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.350008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349805 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo\") pod \"3de3ea20-896e-4605-b13b-302b03e46e75\" (UID: \"3de3ea20-896e-4605-b13b-302b03e46e75\") " Jul 29 06:55:20.350008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349855 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:20.350008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.349970 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 06:55:20.350262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350008 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:20.350262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350179 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.350262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350199 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ea20-896e-4605-b13b-302b03e46e75-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.350262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350214 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.350262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350214 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:20.350917 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.350866 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data" (OuterVolumeSpecName: "istio-data") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:20.352090 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.352069 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr" (OuterVolumeSpecName: "kube-api-access-vnkzr") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "kube-api-access-vnkzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:55:20.352299 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.352281 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 06:55:20.352636 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.352610 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:20.352636 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.352624 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token" (OuterVolumeSpecName: "istio-token") pod "3de3ea20-896e-4605-b13b-302b03e46e75" (UID: "3de3ea20-896e-4605-b13b-302b03e46e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:55:20.450847 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450816 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.450847 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450841 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.450847 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450850 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ea20-896e-4605-b13b-302b03e46e75-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.451062 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450858 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.451062 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450866 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ea20-896e-4605-b13b-302b03e46e75-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.451062 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.450876 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnkzr\" (UniqueName: \"kubernetes.io/projected/3de3ea20-896e-4605-b13b-302b03e46e75-kube-api-access-vnkzr\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:20.755390 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.755341 2556 generic.go:356] "Generic (PLEG): container finished" podID="3de3ea20-896e-4605-b13b-302b03e46e75" containerID="853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e" exitCode=0 Jul 29 06:55:20.755529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.755420 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" Jul 29 06:55:20.755529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.755426 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerDied","Data":"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e"} Jul 29 06:55:20.755529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.755482 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" event={"ID":"3de3ea20-896e-4605-b13b-302b03e46e75","Type":"ContainerDied","Data":"f9296c673a13fee16f93a7b6c97951cf099b086ed34f7487f36fefd0ccf7c59d"} Jul 29 06:55:20.755529 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.755507 2556 scope.go:117] "RemoveContainer" containerID="853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e" Jul 29 06:55:20.764750 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.764734 2556 scope.go:117] "RemoveContainer" containerID="de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b" Jul 29 06:55:20.772484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.772467 2556 scope.go:117] "RemoveContainer" containerID="853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e" Jul 29 06:55:20.772747 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:55:20.772725 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e\": container with ID starting with 853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e not found: ID does not exist" containerID="853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e" Jul 29 06:55:20.772835 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.772751 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e"} err="failed to get container status \"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e\": rpc error: code = NotFound desc = could not find container \"853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e\": container with ID starting with 853c8df53283f6ed4264206e370a58685201cba353024c668379e23e97ee9b7e not found: ID does not exist" Jul 29 06:55:20.772835 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.772767 2556 scope.go:117] "RemoveContainer" containerID="de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b" Jul 29 06:55:20.773004 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:55:20.772986 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b\": container with ID starting with de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b not found: ID does not exist" containerID="de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b" Jul 29 06:55:20.773044 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.773012 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b"} err="failed to get container status \"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b\": rpc error: code = NotFound desc = could not find container \"de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b\": container with ID starting with de25d69a93f14fee582e4b101e2759c51aa86d6faef8cf2e32a190a950482e2b not found: ID does not exist" Jul 29 06:55:20.776521 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.776499 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:55:20.782745 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:20.782724 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9"] Jul 29 06:55:21.155921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:21.155835 2556 patch_prober.go:28] interesting pod/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 06:55:21.155921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:21.155904 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-89063c4d-gw-openshift-default-cf78b87d4-v9sf9" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 06:55:21.926889 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:21.926858 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" path="/var/lib/kubelet/pods/3de3ea20-896e-4605-b13b-302b03e46e75/volumes" Jul 29 06:55:32.775601 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:32.775549 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:53458->10.132.0.33:15021: read: connection reset by peer" start-of-body= Jul 29 06:55:32.775981 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:32.775638 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:53458->10.132.0.33:15021: read: connection reset by peer" Jul 29 06:55:33.815320 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.815284 2556 generic.go:356] "Generic (PLEG): container finished" podID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerID="5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65" exitCode=0 Jul 29 06:55:33.815824 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.815363 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerDied","Data":"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65"} Jul 29 06:55:33.815925 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.815906 2556 scope.go:117] "RemoveContainer" containerID="5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65" Jul 29 06:55:33.816905 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.816877 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:33.817007 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.816945 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:33.817007 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:33.816979 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:34.822668 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:34.822636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerStarted","Data":"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c"} Jul 29 06:55:35.072873 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:35.072778 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:35.073207 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:35.073184 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 29 06:55:35.073354 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:35.073329 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 29 06:55:36.072281 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:36.072239 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 29 06:55:36.072714 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:36.072315 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 29 06:55:37.072639 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:37.072600 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 29 06:55:37.073137 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:37.072668 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 29 06:55:38.072403 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:38.072344 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 29 06:55:38.072602 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:38.072441 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 29 06:55:39.075451 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:39.075421 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:39.075908 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:39.075701 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:39.076347 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:39.076331 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:48.750736 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:48.750699 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:48.818571 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:48.818543 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:48.878221 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:48.878188 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:48.934931 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:48.934906 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:50.092796 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.092758 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:50.093218 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.092948 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" containerID="cri-o://5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c" gracePeriod=30 Jul 29 06:55:50.103729 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.103704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:50.337342 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.337308 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:50.547001 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.546958 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:50.818448 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818287 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818753 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818773 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818810 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818820 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818892 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.822401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.818905 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de3ea20-896e-4605-b13b-302b03e46e75" containerName="istio-proxy" Jul 29 06:55:50.823677 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.823651 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.827319 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.827295 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0f99172f-openshift-default-dockercfg-6pcx7\"" Jul 29 06:55:50.830121 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.830101 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:55:50.839275 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.839250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:50.905015 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.904973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905015 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905020 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lv79\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905057 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905232 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905272 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:50.905618 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:50.905293 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006624 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006666 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006715 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lv79\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006790 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.006861 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.006831 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.007188 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.007099 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.007188 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.007157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.007298 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.007192 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.007364 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.007293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.007693 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.007670 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.009297 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.009277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.009350 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.009325 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.014686 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.014666 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.015937 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.015915 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lv79\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79\") pod \"e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.135589 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.135525 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:51.268438 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.268414 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:55:51.270927 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:55:51.270895 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d35b302_6217_42e7_9ce9_6996ef80d37d.slice/crio-53a374d8935770298d3f59d6b8b724a62f2a1967fbb50ac1f59a5cde9061be0a WatchSource:0}: Error finding container 53a374d8935770298d3f59d6b8b724a62f2a1967fbb50ac1f59a5cde9061be0a: Status 404 returned error can't find the container with id 53a374d8935770298d3f59d6b8b724a62f2a1967fbb50ac1f59a5cde9061be0a Jul 29 06:55:51.273010 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.272984 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:51.273078 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.273047 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:51.273118 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.273076 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:55:51.515402 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.515353 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:51.899797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.899716 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerStarted","Data":"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e"} Jul 29 06:55:51.899797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.899766 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerStarted","Data":"53a374d8935770298d3f59d6b8b724a62f2a1967fbb50ac1f59a5cde9061be0a"} Jul 29 06:55:51.923412 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:51.923351 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podStartSLOduration=1.923336694 podStartE2EDuration="1.923336694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:55:51.921340656 +0000 UTC m=+1996.455420224" watchObservedRunningTime="2026-07-29 06:55:51.923336694 +0000 UTC m=+1996.457416267" Jul 29 06:55:52.136092 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:52.136061 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:52.140759 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:52.140736 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:52.904325 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:52.904288 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:52.905275 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:52.905258 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:55:53.350716 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:53.350685 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:53.430883 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:53.430841 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:53.486619 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:53.486591 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:53.605530 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:53.605443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:53.920431 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:53.920332 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:54.021170 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:54.021139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:55:54.078144 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:54.078110 2556 patch_prober.go:28] interesting pod/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 06:55:54.078298 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:54.078167 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:55:55.342516 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.342490 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:55.452413 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452366 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452424 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452447 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452483 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452515 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452566 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452538 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h7kp\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452823 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452681 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452823 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452726 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452823 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452769 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data\") pod \"d0732e37-fced-4541-ba28-731a2a57e6a9\" (UID: \"d0732e37-fced-4541-ba28-731a2a57e6a9\") " Jul 29 06:55:55.452823 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452805 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:55.453029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452822 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:55.453029 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.452871 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 06:55:55.453139 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453025 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:55.453195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453135 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0732e37-fced-4541-ba28-731a2a57e6a9-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.453195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453157 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.453195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453170 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.453195 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453183 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.453658 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.453542 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:55.454797 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.454770 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:55:55.455100 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.455078 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:55:55.455350 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.455334 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp" (OuterVolumeSpecName: "kube-api-access-2h7kp") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "kube-api-access-2h7kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:55:55.455441 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.455345 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0732e37-fced-4541-ba28-731a2a57e6a9" (UID: "d0732e37-fced-4541-ba28-731a2a57e6a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 06:55:55.553948 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.553919 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.553948 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.553948 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.554126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.553960 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h7kp\" (UniqueName: \"kubernetes.io/projected/d0732e37-fced-4541-ba28-731a2a57e6a9-kube-api-access-2h7kp\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.554126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.553973 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.554126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.553984 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0732e37-fced-4541-ba28-731a2a57e6a9-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:55:55.918884 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.918798 2556 generic.go:356] "Generic (PLEG): container finished" podID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerID="5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c" exitCode=0 Jul 29 06:55:55.918884 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.918845 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" Jul 29 06:55:55.918884 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.918875 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerDied","Data":"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c"} Jul 29 06:55:55.919126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.918932 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw" event={"ID":"d0732e37-fced-4541-ba28-731a2a57e6a9","Type":"ContainerDied","Data":"5a1aa4adfadbfc116695a1b4129e0355954a251ee19acbc5713aec4fe3028a72"} Jul 29 06:55:55.919126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.918966 2556 scope.go:117] "RemoveContainer" containerID="5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c" Jul 29 06:55:55.928723 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.928700 2556 scope.go:117] "RemoveContainer" containerID="5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65" Jul 29 06:55:55.936656 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.936640 2556 scope.go:117] "RemoveContainer" containerID="5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c" Jul 29 06:55:55.936894 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:55:55.936876 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c\": container with ID starting with 5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c not found: ID does not exist" containerID="5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c" Jul 29 06:55:55.936945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.936900 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c"} err="failed to get container status \"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c\": rpc error: code = NotFound desc = could not find container \"5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c\": container with ID starting with 5914068abdb31b0b93453b994127d3a7b5ecc5d7763ad491f0be03410753949c not found: ID does not exist" Jul 29 06:55:55.936945 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.936917 2556 scope.go:117] "RemoveContainer" containerID="5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65" Jul 29 06:55:55.937130 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:55:55.937113 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65\": container with ID starting with 5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65 not found: ID does not exist" containerID="5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65" Jul 29 06:55:55.937171 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.937133 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65"} err="failed to get container status \"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65\": rpc error: code = NotFound desc = could not find container \"5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65\": container with ID starting with 5cfe823f6269f6d8b70cb29ee75f51220a9f2d5b1b1d8f7fd3d5613dfc63cd65 not found: ID does not exist" Jul 29 06:55:55.942067 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.942045 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:55.945612 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:55.945590 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-88aa9015-openshift-default-7cb985f86b-ql4tw"] Jul 29 06:55:57.928961 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:55:57.928921 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" path="/var/lib/kubelet/pods/d0732e37-fced-4541-ba28-731a2a57e6a9/volumes" Jul 29 06:56:05.965168 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.965075 2556 generic.go:356] "Generic (PLEG): container finished" podID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerID="283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e" exitCode=0 Jul 29 06:56:05.965168 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.965151 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerDied","Data":"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e"} Jul 29 06:56:05.965604 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.965540 2556 scope.go:117] "RemoveContainer" containerID="283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e" Jul 29 06:56:05.966523 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.966500 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:56:05.966584 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.966561 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:56:05.966619 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:05.966591 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:56:06.975126 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:06.975088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerStarted","Data":"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263"} Jul 29 06:56:07.136460 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:07.136422 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:07.137689 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:07.137663 2556 patch_prober.go:28] interesting pod/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 06:56:07.137790 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:07.137715 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 06:56:07.905048 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:07.905017 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:08.136482 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:08.136446 2556 patch_prober.go:28] interesting pod/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 06:56:08.136872 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:08.136516 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 06:56:09.136727 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:09.136693 2556 patch_prober.go:28] interesting pod/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 06:56:09.137091 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:09.136747 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 06:56:10.136089 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:10.136057 2556 patch_prober.go:28] interesting pod/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 06:56:10.136262 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:10.136109 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 06:56:11.141416 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:11.141358 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:11.143359 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:11.143334 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:23.989801 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:23.989766 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:24.089719 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.089689 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:24.146154 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.146094 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:24.298265 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.298173 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:56:24.298449 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.298423 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" containerID="cri-o://96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263" gracePeriod=30 Jul 29 06:56:24.324842 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.324792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:24.564531 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:24.564445 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:25.043056 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:25.043023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:25.142629 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:25.142597 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:25.242780 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:25.242748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:26.144518 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:26.144479 2556 patch_prober.go:28] interesting pod/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 06:56:26.144967 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:26.144548 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:56:26.767390 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:26.767354 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:29.550972 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.550950 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:29.649592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649504 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649536 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649570 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lv79\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649592 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649586 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649638 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649709 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649756 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649775 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data\") pod \"5d35b302-6217-42e7-9ce9-6996ef80d37d\" (UID: \"5d35b302-6217-42e7-9ce9-6996ef80d37d\") " Jul 29 06:56:29.649921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.649813 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:56:29.650228 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.650056 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.650228 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.650138 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:56:29.650228 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.650203 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 06:56:29.650583 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.650254 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:56:29.650830 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.650728 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:56:29.651871 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.651849 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 06:56:29.651980 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.651969 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:56:29.652034 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.651967 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79" (OuterVolumeSpecName: "kube-api-access-2lv79") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "kube-api-access-2lv79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:56:29.652400 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.652365 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d35b302-6217-42e7-9ce9-6996ef80d37d" (UID: "5d35b302-6217-42e7-9ce9-6996ef80d37d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:56:29.750703 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750670 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750703 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750697 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750703 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750707 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750703 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750715 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d35b302-6217-42e7-9ce9-6996ef80d37d-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750724 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lv79\" (UniqueName: \"kubernetes.io/projected/5d35b302-6217-42e7-9ce9-6996ef80d37d-kube-api-access-2lv79\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750732 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750744 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d35b302-6217-42e7-9ce9-6996ef80d37d-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:29.750964 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:29.750753 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d35b302-6217-42e7-9ce9-6996ef80d37d-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:56:30.073683 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.073643 2556 generic.go:356] "Generic (PLEG): container finished" podID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerID="96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263" exitCode=0 Jul 29 06:56:30.073858 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.073705 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" Jul 29 06:56:30.073858 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.073687 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerDied","Data":"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263"} Jul 29 06:56:30.073858 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.073825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl" event={"ID":"5d35b302-6217-42e7-9ce9-6996ef80d37d","Type":"ContainerDied","Data":"53a374d8935770298d3f59d6b8b724a62f2a1967fbb50ac1f59a5cde9061be0a"} Jul 29 06:56:30.073858 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.073848 2556 scope.go:117] "RemoveContainer" containerID="96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263" Jul 29 06:56:30.082636 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.082615 2556 scope.go:117] "RemoveContainer" containerID="283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e" Jul 29 06:56:30.091735 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.091660 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:56:30.091803 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.091747 2556 scope.go:117] "RemoveContainer" containerID="96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263" Jul 29 06:56:30.092153 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:56:30.092121 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263\": container with ID starting with 96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263 not found: ID does not exist" containerID="96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263" Jul 29 06:56:30.092295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.092182 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263"} err="failed to get container status \"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263\": rpc error: code = NotFound desc = could not find container \"96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263\": container with ID starting with 96cabe16f79c7faefcb2087ec3af394c706f31e794705b6cf4b05f3ebff38263 not found: ID does not exist" Jul 29 06:56:30.092295 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.092208 2556 scope.go:117] "RemoveContainer" containerID="283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e" Jul 29 06:56:30.092717 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:56:30.092563 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e\": container with ID starting with 283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e not found: ID does not exist" containerID="283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e" Jul 29 06:56:30.092717 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.092593 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e"} err="failed to get container status \"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e\": rpc error: code = NotFound desc = could not find container \"283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e\": container with ID starting with 283093f305d6d65f3051edbbe12e265f7b27a78490bd7be79771c990ff7dbb1e not found: ID does not exist" Jul 29 06:56:30.093981 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:30.093963 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0f99172f-openshift-default-59764db4cb-bqwfl"] Jul 29 06:56:31.927721 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:31.927685 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" path="/var/lib/kubelet/pods/5d35b302-6217-42e7-9ce9-6996ef80d37d/volumes" Jul 29 06:56:32.374824 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:32.374790 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:32.433023 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:32.432979 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:32.529666 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:32.529638 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:32.629724 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:32.629604 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:32.729302 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:32.729252 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:33.161265 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:33.161231 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:50.431259 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:50.431195 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:56:59.095846 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:56:59.095721 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:05.304612 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:05.304575 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:05.410551 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:05.410517 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:05.505322 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:05.505283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:21.380143 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:21.380112 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:26.948921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:26.948885 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:27.049258 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:27.049226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:27.149179 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:27.149144 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:37.724220 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:37.724189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:38.582328 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:38.582264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:44.504349 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:44.504313 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:44.606974 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:44.606943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:44.707608 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:44.707575 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:44.805463 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:44.805348 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:58.566189 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:58.566129 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:58.666271 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:58.666223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:57:58.766311 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:57:58.766274 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:00.527766 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:00.527730 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:06.222394 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:06.222348 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:06.322585 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:06.322551 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:06.866292 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:06.866260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:12.664339 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:12.664307 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:12.722087 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:12.722045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:12.824008 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:12.823949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:13.096011 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.095979 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:58:13.096289 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096277 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096291 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096299 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:58:13.096338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096305 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:58:13.096338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096317 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096338 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096322 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096512 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096391 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096512 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096399 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d35b302-6217-42e7-9ce9-6996ef80d37d" containerName="istio-proxy" Jul 29 06:58:13.096512 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096405 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:58:13.096512 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.096409 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:58:13.099187 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.099170 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.102035 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.102017 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc10550f-openshift-default-dockercfg-52bj5\"" Jul 29 06:58:13.120099 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.120072 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:58:13.122625 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.122604 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:13.202642 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.202607 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.202883 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.202667 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.202883 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.202805 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhg5s\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203025 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203006 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203080 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203134 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203189 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203171 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203243 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203198 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.203243 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.203224 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304639 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304593 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304801 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhg5s\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304801 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304791 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304819 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.304930 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304902 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305094 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305094 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305094 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.304993 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305256 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.305166 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.305333 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.305505 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305649 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.305584 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.305878 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.305858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.307665 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.307639 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.307769 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.307754 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.312654 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.312635 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.312654 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.312642 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhg5s\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s\") pod \"e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.316278 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.316255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:13.409203 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.409129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:13.539613 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.539589 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:58:13.541551 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:58:13.541521 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60de7d4e_d448_437c_81fc_c139dbb0dbe1.slice/crio-1a318b1f3778911545a2a2c37c5dfbccb636780aefa5bbe00236ebedfaf527de WatchSource:0}: Error finding container 1a318b1f3778911545a2a2c37c5dfbccb636780aefa5bbe00236ebedfaf527de: Status 404 returned error can't find the container with id 1a318b1f3778911545a2a2c37c5dfbccb636780aefa5bbe00236ebedfaf527de Jul 29 06:58:13.543563 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.543532 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:13.543655 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.543597 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:13.543655 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:13.543621 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:14.538495 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:14.538451 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerStarted","Data":"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088"} Jul 29 06:58:14.538910 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:14.538539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerStarted","Data":"1a318b1f3778911545a2a2c37c5dfbccb636780aefa5bbe00236ebedfaf527de"} Jul 29 06:58:14.559416 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:14.559348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podStartSLOduration=1.559332642 podStartE2EDuration="1.559332642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:58:14.556942544 +0000 UTC m=+2139.091022135" watchObservedRunningTime="2026-07-29 06:58:14.559332642 +0000 UTC m=+2139.093412214" Jul 29 06:58:15.410059 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:15.410018 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:15.414997 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:15.414968 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:15.543502 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:15.543471 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:15.544477 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:15.544460 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:16.541119 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:16.541079 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:16.614427 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:16.614371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:16.683921 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:16.683887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:17.117151 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:17.117096 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:17.221793 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:17.221725 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:17.321141 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:17.321096 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:31.325600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.325572 2556 patch_prober.go:28] interesting pod/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:59628->10.132.0.35:15021: read: connection reset by peer" start-of-body= Jul 29 06:58:31.326006 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.325633 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:59628->10.132.0.35:15021: read: connection reset by peer" Jul 29 06:58:31.615686 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.615582 2556 generic.go:356] "Generic (PLEG): container finished" podID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerID="7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088" exitCode=0 Jul 29 06:58:31.615686 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.615639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerDied","Data":"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088"} Jul 29 06:58:31.616006 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.615992 2556 scope.go:117] "RemoveContainer" containerID="7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088" Jul 29 06:58:31.617026 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.616998 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:31.617093 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.617064 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:31.617139 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:31.617096 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:58:32.621595 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:32.621560 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerStarted","Data":"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814"} Jul 29 06:58:33.409708 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:33.409667 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:33.409916 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:33.409871 2556 patch_prober.go:28] interesting pod/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 29 06:58:33.409994 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:33.409926 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 29 06:58:34.410542 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:34.410503 2556 patch_prober.go:28] interesting pod/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 29 06:58:34.411015 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:34.410586 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 29 06:58:35.409749 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:35.409709 2556 patch_prober.go:28] interesting pod/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 29 06:58:35.409924 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:35.409775 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 29 06:58:36.413599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:36.413571 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:36.413993 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:36.413817 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:36.414400 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:36.414359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:58:41.460281 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:41.460249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:42.715401 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:42.715346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:48.183605 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:48.183563 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:48.283599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:48.283569 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:58.968230 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:58.968185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:59.067805 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:59.067734 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:59.122036 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:59.122004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:58:59.480233 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:58:59.480193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.134415 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.134335 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.191051 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.191022 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.377804 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.377765 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:59:05.378030 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.377985 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" containerID="cri-o://f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814" gracePeriod=30 Jul 29 06:59:05.399691 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.399602 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.624283 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.624247 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.738929 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.738900 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.837463 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.837431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:05.949348 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:05.949316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:06.416192 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:06.416162 2556 patch_prober.go:28] interesting pod/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 06:59:06.416569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:06.416242 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 06:59:10.616674 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.616647 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:59:10.775983 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.775954 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776006 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776021 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776056 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776076 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776105 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776140 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776125 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776157 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776186 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhg5s\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s\") pod \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\" (UID: \"60de7d4e-d448-437c-81fc-c139dbb0dbe1\") " Jul 29 06:59:10.776484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776372 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:59:10.776484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:59:10.776484 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776469 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 06:59:10.776727 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.776596 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:59:10.777333 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.777304 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data" (OuterVolumeSpecName: "istio-data") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:59:10.778339 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.778314 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 06:59:10.778530 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.778505 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s" (OuterVolumeSpecName: "kube-api-access-lhg5s") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "kube-api-access-lhg5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:59:10.778631 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.778563 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 06:59:10.778685 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.778651 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token" (OuterVolumeSpecName: "istio-token") pod "60de7d4e-d448-437c-81fc-c139dbb0dbe1" (UID: "60de7d4e-d448-437c-81fc-c139dbb0dbe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 06:59:10.786923 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.786899 2556 generic.go:356] "Generic (PLEG): container finished" podID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerID="f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814" exitCode=0 Jul 29 06:59:10.787017 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.786968 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" Jul 29 06:59:10.787017 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.786978 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerDied","Data":"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814"} Jul 29 06:59:10.787091 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.787034 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7" event={"ID":"60de7d4e-d448-437c-81fc-c139dbb0dbe1","Type":"ContainerDied","Data":"1a318b1f3778911545a2a2c37c5dfbccb636780aefa5bbe00236ebedfaf527de"} Jul 29 06:59:10.787091 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.787056 2556 scope.go:117] "RemoveContainer" containerID="f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814" Jul 29 06:59:10.798662 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.798648 2556 scope.go:117] "RemoveContainer" containerID="7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088" Jul 29 06:59:10.806022 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.806000 2556 scope.go:117] "RemoveContainer" containerID="f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814" Jul 29 06:59:10.806253 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:59:10.806234 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814\": container with ID starting with f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814 not found: ID does not exist" containerID="f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814" Jul 29 06:59:10.806317 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.806260 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814"} err="failed to get container status \"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814\": rpc error: code = NotFound desc = could not find container \"f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814\": container with ID starting with f2dbf488d2d9249ee9b565c2cf5a13e94b2cbeb8dc3eba723e27918f54cc6814 not found: ID does not exist" Jul 29 06:59:10.806317 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.806277 2556 scope.go:117] "RemoveContainer" containerID="7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088" Jul 29 06:59:10.806550 ip-10-0-131-31 kubenswrapper[2556]: E0729 06:59:10.806533 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088\": container with ID starting with 7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088 not found: ID does not exist" containerID="7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088" Jul 29 06:59:10.806599 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.806556 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088"} err="failed to get container status \"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088\": rpc error: code = NotFound desc = could not find container \"7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088\": container with ID starting with 7ac2d71722cac13c6f8c310e70f24fdf8aa1b105a19b5e42eaae2257f95b7088 not found: ID does not exist" Jul 29 06:59:10.810430 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.810411 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:59:10.814207 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.814187 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc10550f-openshift-default-577c7c96d4-ctzs7"] Jul 29 06:59:10.877028 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877006 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhg5s\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-kube-api-access-lhg5s\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877028 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877026 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877036 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877043 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877052 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877062 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877075 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877083 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60de7d4e-d448-437c-81fc-c139dbb0dbe1-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:10.877133 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:10.877091 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60de7d4e-d448-437c-81fc-c139dbb0dbe1-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 06:59:11.680996 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:11.680957 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:11.742333 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:11.742305 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:11.822980 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:11.822951 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:11.915287 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:11.915260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:11.932428 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:11.927864 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" path="/var/lib/kubelet/pods/60de7d4e-d448-437c-81fc-c139dbb0dbe1/volumes" Jul 29 06:59:12.053814 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:12.053779 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:17.632389 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:17.632347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:17.733796 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:17.733754 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:24.506884 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.506845 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:24.871325 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871238 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 06:59:24.871572 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871560 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.871616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871574 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.871616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871585 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:59:24.871616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871591 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0732e37-fced-4541-ba28-731a2a57e6a9" containerName="istio-proxy" Jul 29 06:59:24.871616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871606 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.871616 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871612 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.871772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871661 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.871772 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.871667 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="60de7d4e-d448-437c-81fc-c139dbb0dbe1" containerName="istio-proxy" Jul 29 06:59:24.875811 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.875795 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.878959 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.878936 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-11cbe871-openshift-default-dockercfg-c25hf\"" Jul 29 06:59:24.897324 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.897297 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 06:59:24.909848 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.909826 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:24.996139 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996103 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996315 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996150 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996315 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbtx\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996315 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996293 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996315 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996406 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996474 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:24.996569 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:24.996493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097322 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097358 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097439 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097621 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097578 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbtx\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097958 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097782 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097958 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.097958 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.097922 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.098547 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.098501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.101092 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.098772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.105714 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.105682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.106054 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.106029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.108083 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.108058 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.108176 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.108131 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbtx\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx\") pod \"e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.186208 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.186131 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:25.188669 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.188625 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:25.321660 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.321634 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 06:59:25.324283 ip-10-0-131-31 kubenswrapper[2556]: W0729 06:59:25.324259 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068d93bb_c599_464d_a9a1_1fb4fc7595fb.slice/crio-40552be892d56c37d08f87183d56a318e1388bcb2da0ec5ab54a0c4507e4d1d1 WatchSource:0}: Error finding container 40552be892d56c37d08f87183d56a318e1388bcb2da0ec5ab54a0c4507e4d1d1: Status 404 returned error can't find the container with id 40552be892d56c37d08f87183d56a318e1388bcb2da0ec5ab54a0c4507e4d1d1 Jul 29 06:59:25.326441 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.326408 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:25.326544 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.326485 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:25.326544 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.326529 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:25.857734 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.857701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerStarted","Data":"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77"} Jul 29 06:59:25.858128 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.857754 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerStarted","Data":"40552be892d56c37d08f87183d56a318e1388bcb2da0ec5ab54a0c4507e4d1d1"} Jul 29 06:59:25.878761 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:25.878707 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podStartSLOduration=1.8786933220000002 podStartE2EDuration="1.878693322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:59:25.876942517 +0000 UTC m=+2210.411022092" watchObservedRunningTime="2026-07-29 06:59:25.878693322 +0000 UTC m=+2210.412772938" Jul 29 06:59:26.186805 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:26.186722 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:26.188204 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:26.188179 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:26.188329 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:26.188231 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:27.191124 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.191089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:27.764095 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.764063 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:27.861157 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.861117 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:27.866794 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.866772 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:27.867818 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.867800 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:27.997120 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:27.997088 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:28.098352 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:28.098243 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:28.456617 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:28.456587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:28.556096 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:28.556060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:38.919610 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:38.919579 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:42.868680 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.868645 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:42.869097 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.868705 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:42.940506 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.940474 2556 generic.go:356] "Generic (PLEG): container finished" podID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerID="2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77" exitCode=0 Jul 29 06:59:42.940662 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.940531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerDied","Data":"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77"} Jul 29 06:59:42.940838 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.940824 2556 scope.go:117] "RemoveContainer" containerID="2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77" Jul 29 06:59:42.941821 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.941796 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:42.941883 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.941862 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:42.941922 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:42.941892 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 06:59:43.946766 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:43.946735 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerStarted","Data":"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894"} Jul 29 06:59:44.187194 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:44.187156 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:44.187501 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:44.187474 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:44.187600 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:44.187534 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:45.187179 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:45.187140 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:45.187591 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:45.187228 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:46.187112 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:46.187074 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:46.187291 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:46.187140 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:47.186852 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:47.186815 2556 patch_prober.go:28] interesting pod/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 06:59:47.187037 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:47.186890 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 06:59:48.190798 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:48.190766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:48.191221 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:48.191047 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:48.191650 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:48.191636 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 06:59:52.750350 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:52.750319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 06:59:54.649290 ip-10-0-131-31 kubenswrapper[2556]: I0729 06:59:54.649253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:02.657982 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:02.657865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:05.001358 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.001316 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:00:05.004070 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.004049 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.008263 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.008234 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11cbe871\"/\"kube-root-ca.crt\"" Jul 29 07:00:05.008263 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.008256 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11cbe871\"/\"e2e-lifecycle-model-11cbe871-kserve-self-signed-certs\"" Jul 29 07:00:05.008470 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.008242 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-11cbe871\"/\"openshift-service-ca.crt\"" Jul 29 07:00:05.008470 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.008297 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-11cbe871\"/\"default-dockercfg-4hmvb\"" Jul 29 07:00:05.014844 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.014825 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:00:05.023090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.023053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:05.143768 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.143930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.143930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143799 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.143930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq55h\" (UniqueName: \"kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.143930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143865 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.144067 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.143950 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245002 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.244966 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq55h\" (UniqueName: \"kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245002 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245234 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245234 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245234 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245135 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245234 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245481 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245460 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245536 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245484 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.245580 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.245555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.247560 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.247532 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.247821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.247804 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.252898 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.252828 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq55h\" (UniqueName: \"kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h\") pod \"e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.313462 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.313434 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:05.441327 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:05.441305 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:00:05.443046 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:00:05.443019 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaf936a8_7ab2_4002_804c_34fd73ed3908.slice/crio-f894a437fe6ad84ab202ec8926a993c10af5a9c465724bc4731e43893070aff4 WatchSource:0}: Error finding container f894a437fe6ad84ab202ec8926a993c10af5a9c465724bc4731e43893070aff4: Status 404 returned error can't find the container with id f894a437fe6ad84ab202ec8926a993c10af5a9c465724bc4731e43893070aff4 Jul 29 07:00:06.068850 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:06.068809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" event={"ID":"daf936a8-7ab2-4002-804c-34fd73ed3908","Type":"ContainerStarted","Data":"b42d19825ef7f9875eb98bc118c764c823921d9a2698121a9348276e533b455b"} Jul 29 07:00:06.069258 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:06.068859 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" event={"ID":"daf936a8-7ab2-4002-804c-34fd73ed3908","Type":"ContainerStarted","Data":"f894a437fe6ad84ab202ec8926a993c10af5a9c465724bc4731e43893070aff4"} Jul 29 07:00:06.087112 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:06.087063 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" podStartSLOduration=2.087047582 podStartE2EDuration="2.087047582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:00:06.085776834 +0000 UTC m=+2250.619856404" watchObservedRunningTime="2026-07-29 07:00:06.087047582 +0000 UTC m=+2250.621127154" Jul 29 07:00:15.314021 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:15.313986 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:15.314021 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:15.314032 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:15.326499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:15.326467 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:16.123744 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:16.123713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:00:42.807685 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:42.807647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:42.905595 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:42.905562 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:43.005952 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:43.005920 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:46.204667 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:46.204614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:00:51.188855 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:51.188816 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:00:51.284634 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:00:51.284595 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:06.207945 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.207888 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" podUID="daf936a8-7ab2-4002-804c-34fd73ed3908" containerName="main" containerID="cri-o://b42d19825ef7f9875eb98bc118c764c823921d9a2698121a9348276e533b455b" gracePeriod=45 Jul 29 07:01:06.337560 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.337516 2556 generic.go:356] "Generic (PLEG): container finished" podID="daf936a8-7ab2-4002-804c-34fd73ed3908" containerID="b42d19825ef7f9875eb98bc118c764c823921d9a2698121a9348276e533b455b" exitCode=0 Jul 29 07:01:06.337706 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.337577 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" event={"ID":"daf936a8-7ab2-4002-804c-34fd73ed3908","Type":"ContainerDied","Data":"b42d19825ef7f9875eb98bc118c764c823921d9a2698121a9348276e533b455b"} Jul 29 07:01:06.457220 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.457191 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:01:06.570661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570570 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.570661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570619 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.570661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.570947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570691 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.570947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570725 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq55h\" (UniqueName: \"kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.570947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570764 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home\") pod \"daf936a8-7ab2-4002-804c-34fd73ed3908\" (UID: \"daf936a8-7ab2-4002-804c-34fd73ed3908\") " Jul 29 07:01:06.571102 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570977 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache" (OuterVolumeSpecName: "model-cache") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:06.571102 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.570993 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:06.571102 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.571076 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home" (OuterVolumeSpecName: "home") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:06.571223 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.571102 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:06.572894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.572874 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm" (OuterVolumeSpecName: "dshm") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:06.573004 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.572958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:01:06.573050 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.573026 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h" (OuterVolumeSpecName: "kube-api-access-gq55h") pod "daf936a8-7ab2-4002-804c-34fd73ed3908" (UID: "daf936a8-7ab2-4002-804c-34fd73ed3908"). InnerVolumeSpecName "kube-api-access-gq55h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:01:06.672489 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.672450 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq55h\" (UniqueName: \"kubernetes.io/projected/daf936a8-7ab2-4002-804c-34fd73ed3908-kube-api-access-gq55h\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:06.672489 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.672479 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:06.672489 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.672491 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daf936a8-7ab2-4002-804c-34fd73ed3908-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:06.672722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.672502 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:06.672722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:06.672513 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf936a8-7ab2-4002-804c-34fd73ed3908-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:07.343625 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.343595 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" Jul 29 07:01:07.344093 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.343596 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l" event={"ID":"daf936a8-7ab2-4002-804c-34fd73ed3908","Type":"ContainerDied","Data":"f894a437fe6ad84ab202ec8926a993c10af5a9c465724bc4731e43893070aff4"} Jul 29 07:01:07.344093 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.343721 2556 scope.go:117] "RemoveContainer" containerID="b42d19825ef7f9875eb98bc118c764c823921d9a2698121a9348276e533b455b" Jul 29 07:01:07.364571 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.364280 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:01:07.366085 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.366066 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-11cbe871/e2e-lifecycle-model-11cbe871-kserve-5f774fb8b7-ccz4l"] Jul 29 07:01:07.927241 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:07.927211 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf936a8-7ab2-4002-804c-34fd73ed3908" path="/var/lib/kubelet/pods/daf936a8-7ab2-4002-804c-34fd73ed3908/volumes" Jul 29 07:01:27.721429 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:27.721322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:28.108399 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:28.108256 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:33.585452 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:33.585414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:33.656429 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:33.656365 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:33.829239 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:33.829207 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 07:01:33.829475 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:33.829437 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" containerID="cri-o://6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894" gracePeriod=30 Jul 29 07:01:33.851226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:33.851160 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:34.140651 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:34.140511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:35.130873 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:35.130842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:35.225584 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:35.225550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:35.330807 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:35.330780 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:36.090748 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:36.090715 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:36.189240 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:36.189206 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:36.289891 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:36.289863 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:37.717220 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:37.717186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:39.091496 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.091437 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 07:01:39.153340 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153312 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153350 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153388 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153424 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153455 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbtx\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153471 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153489 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153547 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153623 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data\") pod \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\" (UID: \"068d93bb-c599-464d-a9a1-1fb4fc7595fb\") " Jul 29 07:01:39.153827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153624 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:39.153827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153768 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:39.153827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153800 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:01:39.154092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.153972 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:39.154092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.154018 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.154092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.154037 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.154092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.154054 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.154599 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.154577 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:39.155801 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.155774 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:01:39.155920 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.155866 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx" (OuterVolumeSpecName: "kube-api-access-hrbtx") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "kube-api-access-hrbtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:01:39.156317 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.156292 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:01:39.156440 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.156320 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "068d93bb-c599-464d-a9a1-1fb4fc7595fb" (UID: "068d93bb-c599-464d-a9a1-1fb4fc7595fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:01:39.254743 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254703 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbtx\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-kube-api-access-hrbtx\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.254743 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254737 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.254743 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254747 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.254743 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254760 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.255019 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254767 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.255019 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.254775 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068d93bb-c599-464d-a9a1-1fb4fc7595fb-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:01:39.486336 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.486300 2556 generic.go:356] "Generic (PLEG): container finished" podID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerID="6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894" exitCode=0 Jul 29 07:01:39.486519 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.486371 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" Jul 29 07:01:39.486519 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.486407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerDied","Data":"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894"} Jul 29 07:01:39.486519 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.486472 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm" event={"ID":"068d93bb-c599-464d-a9a1-1fb4fc7595fb","Type":"ContainerDied","Data":"40552be892d56c37d08f87183d56a318e1388bcb2da0ec5ab54a0c4507e4d1d1"} Jul 29 07:01:39.486519 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.486502 2556 scope.go:117] "RemoveContainer" containerID="6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894" Jul 29 07:01:39.495454 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.495437 2556 scope.go:117] "RemoveContainer" containerID="2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77" Jul 29 07:01:39.503188 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.503168 2556 scope.go:117] "RemoveContainer" containerID="6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894" Jul 29 07:01:39.503482 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:01:39.503458 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894\": container with ID starting with 6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894 not found: ID does not exist" containerID="6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894" Jul 29 07:01:39.503575 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.503493 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894"} err="failed to get container status \"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894\": rpc error: code = NotFound desc = could not find container \"6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894\": container with ID starting with 6ddeaf17bff9960a62567e2c4c1f6592532b506b7736f6be40cd3e61f1b79894 not found: ID does not exist" Jul 29 07:01:39.503575 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.503514 2556 scope.go:117] "RemoveContainer" containerID="2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77" Jul 29 07:01:39.503803 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:01:39.503784 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77\": container with ID starting with 2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77 not found: ID does not exist" containerID="2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77" Jul 29 07:01:39.503853 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.503811 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77"} err="failed to get container status \"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77\": rpc error: code = NotFound desc = could not find container \"2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77\": container with ID starting with 2e9c9e1eb3757f48fe910635fdcc667ee4189385eb07f0213d66a36a21797e77 not found: ID does not exist" Jul 29 07:01:39.507964 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.507940 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 07:01:39.512850 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.512829 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11cbe871-openshift-default-cd6865c76-vtvxm"] Jul 29 07:01:39.927321 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:39.927238 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" path="/var/lib/kubelet/pods/068d93bb-c599-464d-a9a1-1fb4fc7595fb/volumes" Jul 29 07:01:43.340119 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:43.340082 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:43.525187 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:43.525157 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:44.037840 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:44.037806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:49.875998 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:49.875956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:49.983415 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:49.983343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:01:50.075628 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:01:50.075592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:00.567976 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:00.567933 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:00.892148 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:00.892021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:17.201152 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:17.201115 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:17.301358 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:17.301325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:21.840547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:21.840514 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:27.923390 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:27.923338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:28.024317 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.024285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:28.234844 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.234806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:28.516831 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.516757 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:02:28.517065 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517054 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.517108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517067 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.517108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517075 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf936a8-7ab2-4002-804c-34fd73ed3908" containerName="main" Jul 29 07:02:28.517108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517080 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf936a8-7ab2-4002-804c-34fd73ed3908" containerName="main" Jul 29 07:02:28.517108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517089 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.517108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517095 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.517264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517151 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.517264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517157 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf936a8-7ab2-4002-804c-34fd73ed3908" containerName="main" Jul 29 07:02:28.517264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.517164 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="068d93bb-c599-464d-a9a1-1fb4fc7595fb" containerName="istio-proxy" Jul 29 07:02:28.519910 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.519894 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.522897 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.522873 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dd1393-openshift-default-dockercfg-gdgpl\"" Jul 29 07:02:28.533599 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.533578 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:02:28.541319 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.541295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:28.573432 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573409 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573573 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsdn\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573648 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573648 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573602 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573755 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573755 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573857 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573857 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573804 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.573857 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.573833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsdn\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675351 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675626 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675651 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675719 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.675794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.675761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.676106 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.676088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.677852 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.677833 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.678066 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.678048 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.683201 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.683171 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.683323 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.683280 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsdn\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn\") pod \"e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.829679 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.829583 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:28.911708 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.911671 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:28.961627 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.961591 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:02:28.964644 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:02:28.964608 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c317822_e7d0_42c2_a3d2_68d5f5b55c40.slice/crio-c60e5f2cfacc8617de1cf9067ed4a0b3a69ad45b45597bb7ad41a8db9780d2cd WatchSource:0}: Error finding container c60e5f2cfacc8617de1cf9067ed4a0b3a69ad45b45597bb7ad41a8db9780d2cd: Status 404 returned error can't find the container with id c60e5f2cfacc8617de1cf9067ed4a0b3a69ad45b45597bb7ad41a8db9780d2cd Jul 29 07:02:28.966907 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.966871 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:28.967224 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.967197 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:28.967369 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:28.967256 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:29.702430 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:29.702361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerStarted","Data":"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459"} Jul 29 07:02:29.702631 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:29.702508 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerStarted","Data":"c60e5f2cfacc8617de1cf9067ed4a0b3a69ad45b45597bb7ad41a8db9780d2cd"} Jul 29 07:02:29.722912 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:29.722858 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podStartSLOduration=1.722843235 podStartE2EDuration="1.722843235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:02:29.721794303 +0000 UTC m=+2394.255873876" watchObservedRunningTime="2026-07-29 07:02:29.722843235 +0000 UTC m=+2394.256922807" Jul 29 07:02:29.830554 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:29.830519 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:30.834538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:30.834511 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:31.587500 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.587463 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:31.675064 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.675027 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:31.710337 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.710309 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:31.711212 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.711192 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:31.742040 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.742004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:31.939101 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:31.939077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:32.358281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:32.358185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:32.422119 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:32.422082 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:46.711347 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.711310 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:46.711772 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.711373 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:46.790450 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.790417 2556 generic.go:356] "Generic (PLEG): container finished" podID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerID="6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459" exitCode=0 Jul 29 07:02:46.790614 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.790493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerDied","Data":"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459"} Jul 29 07:02:46.790889 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.790873 2556 scope.go:117] "RemoveContainer" containerID="6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459" Jul 29 07:02:46.791896 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.791865 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:46.792001 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.791952 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:46.792062 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:46.792003 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:47.797129 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:47.797092 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerStarted","Data":"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c"} Jul 29 07:02:47.829886 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:47.829857 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:47.830039 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:47.830019 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:47.830090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:47.830071 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:48.830759 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:48.830718 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:48.831145 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:48.830782 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:49.830710 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:49.830677 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:49.830894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:49.830728 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:50.830305 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:50.830270 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:50.830491 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:50.830321 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:51.830434 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:51.830401 2556 patch_prober.go:28] interesting pod/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 07:02:51.830805 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:51.830456 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 07:02:52.834423 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:52.834393 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:52.834783 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:52.834653 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:52.835285 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:52.835266 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:02:56.645155 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:56.645075 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:56.893217 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:56.893187 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:57.154920 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.154883 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:02:57.157347 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.157325 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.160264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.160242 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9aa496-openshift-default-dockercfg-2zhkw\"" Jul 29 07:02:57.168092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.168068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:02:57.181687 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.181658 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:57.241746 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241708 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.241918 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241759 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lq9l\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.241918 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.241918 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.241918 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241887 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.242058 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.242058 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.242058 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.242058 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.241993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.342942 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.342906 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lq9l\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.342942 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.342952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.342983 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343056 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343104 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343492 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343492 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343459 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343589 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343517 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343650 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343875 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343700 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.343930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.343901 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.345369 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.345346 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.345675 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.345657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.351804 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.351783 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.351972 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.351953 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lq9l\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l\") pod \"e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.467805 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.467769 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:57.601000 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.600976 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:02:57.602835 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:02:57.602807 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf0a136_6257_4c64_b389_e43bce21aa4f.slice/crio-00ddb362a12a159429807d15ed224350ff2624146e1d363724e1455d7c9aa2db WatchSource:0}: Error finding container 00ddb362a12a159429807d15ed224350ff2624146e1d363724e1455d7c9aa2db: Status 404 returned error can't find the container with id 00ddb362a12a159429807d15ed224350ff2624146e1d363724e1455d7c9aa2db Jul 29 07:02:57.605185 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.605152 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:57.605267 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.605216 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:57.605267 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.605243 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:02:57.840886 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.840784 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerStarted","Data":"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb"} Jul 29 07:02:57.840886 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.840877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerStarted","Data":"00ddb362a12a159429807d15ed224350ff2624146e1d363724e1455d7c9aa2db"} Jul 29 07:02:57.859044 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.858992 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podStartSLOduration=0.858976801 podStartE2EDuration="858.976801ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:02:57.85788623 +0000 UTC m=+2422.391965803" watchObservedRunningTime="2026-07-29 07:02:57.858976801 +0000 UTC m=+2422.393056373" Jul 29 07:02:57.974671 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:57.974630 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:58.468761 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:58.468722 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:58.473958 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:58.473935 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:58.845901 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:58.845798 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:58.846837 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:58.846816 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:02:59.753673 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:59.753638 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:02:59.835008 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:02:59.834969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:00.004770 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:00.004678 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:00.172121 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:00.172089 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:00.421756 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:00.421673 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:00.512843 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:00.512809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:14.887566 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:14.887515 2556 patch_prober.go:28] interesting pod/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:03:14.888162 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:14.887600 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:03:15.928325 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.928290 2556 generic.go:356] "Generic (PLEG): container finished" podID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerID="ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb" exitCode=0 Jul 29 07:03:15.928873 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.928369 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerDied","Data":"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb"} Jul 29 07:03:15.928873 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.928851 2556 scope.go:117] "RemoveContainer" containerID="ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb" Jul 29 07:03:15.929899 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.929872 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:03:15.930001 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.929937 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:03:15.930001 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:15.929970 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:03:16.935038 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:16.935002 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerStarted","Data":"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac"} Jul 29 07:03:17.468696 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:17.468613 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:03:17.468925 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:17.468892 2556 patch_prober.go:28] interesting pod/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 29 07:03:17.469012 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:17.468958 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 29 07:03:18.468786 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:18.468750 2556 patch_prober.go:28] interesting pod/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 29 07:03:18.469407 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:18.469352 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 29 07:03:19.468500 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:19.468456 2556 patch_prober.go:28] interesting pod/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 29 07:03:19.468708 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:19.468525 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 29 07:03:20.468604 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:20.468559 2556 patch_prober.go:28] interesting pod/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 29 07:03:20.469108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:20.468627 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 29 07:03:21.471701 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:21.471671 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:03:21.472081 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:21.472048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:03:21.472748 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:21.472730 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:03:24.723400 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:24.723351 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:03:43.076839 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:03:43.076781 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:04:11.602465 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.602432 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:04:11.604911 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.604884 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.609137 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.609112 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9aa496\"/\"default-dockercfg-wrld4\"" Jul 29 07:04:11.609268 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.609117 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9aa496\"/\"test-model-9aa496-kserve-self-signed-certs\"" Jul 29 07:04:11.609268 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.609157 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9aa496\"/\"openshift-service-ca.crt\"" Jul 29 07:04:11.609268 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.609163 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9aa496\"/\"kube-root-ca.crt\"" Jul 29 07:04:11.613816 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.613794 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:04:11.629235 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.629204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:04:11.701438 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.701572 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701445 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.701572 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.701572 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701519 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.701712 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701605 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2kk\" (UniqueName: \"kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.701712 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.701673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802367 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802338 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802503 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802371 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802503 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802602 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802511 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802658 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2kk\" (UniqueName: \"kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802658 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802763 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802878 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802857 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.802938 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.802907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.804825 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.804805 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.805077 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.805058 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.810641 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.810615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2kk\" (UniqueName: \"kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk\") pod \"test-model-9aa496-kserve-787bb66bf7-gpn5k\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:11.916056 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:11.915972 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:12.049843 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:12.049824 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:04:12.052279 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:04:12.052247 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8992f04b_e9cc_4235_816d_b594adb54f67.slice/crio-9ae9d5c5b3c8cbf474535bc4ca303184ac91d47df9ec94812b14ea662d039f1c WatchSource:0}: Error finding container 9ae9d5c5b3c8cbf474535bc4ca303184ac91d47df9ec94812b14ea662d039f1c: Status 404 returned error can't find the container with id 9ae9d5c5b3c8cbf474535bc4ca303184ac91d47df9ec94812b14ea662d039f1c Jul 29 07:04:12.178937 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:12.178851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" event={"ID":"8992f04b-e9cc-4235-816d-b594adb54f67","Type":"ContainerStarted","Data":"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5"} Jul 29 07:04:12.178937 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:12.178901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" event={"ID":"8992f04b-e9cc-4235-816d-b594adb54f67","Type":"ContainerStarted","Data":"9ae9d5c5b3c8cbf474535bc4ca303184ac91d47df9ec94812b14ea662d039f1c"} Jul 29 07:04:12.195111 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:12.195060 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" podStartSLOduration=1.1950393830000001 podStartE2EDuration="1.195039383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:04:12.194310855 +0000 UTC m=+2496.728390428" watchObservedRunningTime="2026-07-29 07:04:12.195039383 +0000 UTC m=+2496.729118958" Jul 29 07:04:21.916296 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:21.916254 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:21.916793 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:21.916311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:21.928869 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:21.928846 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:22.236793 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:22.236760 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:04:51.640464 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:04:51.640353 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:09.916343 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:09.916302 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:10.830693 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:10.830663 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:05:10.847931 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:10.847900 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:21.376029 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:21.375993 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:21.435452 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:21.435422 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:21.491916 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:21.491870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:21.553236 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:21.553190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:21.907822 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:21.907789 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:25.850161 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:25.850094 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" podUID="8992f04b-e9cc-4235-816d-b594adb54f67" containerName="main" containerID="cri-o://fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5" gracePeriod=45 Jul 29 07:05:26.094328 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.094299 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:05:26.155620 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155545 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x2kk\" (UniqueName: \"kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.155620 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155602 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.155821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155648 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.155821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155695 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.155821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155736 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.155821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155792 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir\") pod \"8992f04b-e9cc-4235-816d-b594adb54f67\" (UID: \"8992f04b-e9cc-4235-816d-b594adb54f67\") " Jul 29 07:05:26.156018 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.155861 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache" (OuterVolumeSpecName: "model-cache") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:26.156071 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.156042 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home" (OuterVolumeSpecName: "home") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:26.156071 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.156055 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:26.156165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.156115 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.156165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.156127 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.157898 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.157877 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:05:26.158306 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.158290 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk" (OuterVolumeSpecName: "kube-api-access-9x2kk") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "kube-api-access-9x2kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:05:26.158350 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.158289 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm" (OuterVolumeSpecName: "dshm") pod "8992f04b-e9cc-4235-816d-b594adb54f67" (UID: "8992f04b-e9cc-4235-816d-b594adb54f67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:26.257233 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.257205 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.257233 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.257229 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8992f04b-e9cc-4235-816d-b594adb54f67-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.257233 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.257240 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2kk\" (UniqueName: \"kubernetes.io/projected/8992f04b-e9cc-4235-816d-b594adb54f67-kube-api-access-9x2kk\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.257547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.257249 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8992f04b-e9cc-4235-816d-b594adb54f67-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:26.515355 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.515317 2556 generic.go:356] "Generic (PLEG): container finished" podID="8992f04b-e9cc-4235-816d-b594adb54f67" containerID="fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5" exitCode=0 Jul 29 07:05:26.515567 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.515395 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" Jul 29 07:05:26.515567 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.515406 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" event={"ID":"8992f04b-e9cc-4235-816d-b594adb54f67","Type":"ContainerDied","Data":"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5"} Jul 29 07:05:26.515567 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.515452 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k" event={"ID":"8992f04b-e9cc-4235-816d-b594adb54f67","Type":"ContainerDied","Data":"9ae9d5c5b3c8cbf474535bc4ca303184ac91d47df9ec94812b14ea662d039f1c"} Jul 29 07:05:26.515567 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.515468 2556 scope.go:117] "RemoveContainer" containerID="fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5" Jul 29 07:05:26.525976 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.525954 2556 scope.go:117] "RemoveContainer" containerID="fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5" Jul 29 07:05:26.526532 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:05:26.526510 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5\": container with ID starting with fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5 not found: ID does not exist" containerID="fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5" Jul 29 07:05:26.526626 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.526542 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5"} err="failed to get container status \"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5\": rpc error: code = NotFound desc = could not find container \"fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5\": container with ID starting with fb84a19c15c6b88236b5ca4eb637c27833c4e50157d9701b6576c987def580c5 not found: ID does not exist" Jul 29 07:05:26.538741 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.538720 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:05:26.543731 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:26.543709 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9aa496/test-model-9aa496-kserve-787bb66bf7-gpn5k"] Jul 29 07:05:27.461864 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.461831 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:27.560392 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.560342 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:27.699406 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.699359 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:05:27.699650 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.699629 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" containerID="cri-o://8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c" gracePeriod=30 Jul 29 07:05:27.719049 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.718985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:27.927625 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.927591 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8992f04b-e9cc-4235-816d-b594adb54f67" path="/var/lib/kubelet/pods/8992f04b-e9cc-4235-816d-b594adb54f67/volumes" Jul 29 07:05:27.938174 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:27.938146 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:28.315413 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:28.315369 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:28.415759 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:28.415727 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:32.946154 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:32.946131 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:05:33.013802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013725 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.013802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013788 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013813 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013854 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013882 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013952 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlsdn\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.013982 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014013 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014409 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014037 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert\") pod \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\" (UID: \"2c317822-e7d0-42c2-a3d2-68d5f5b55c40\") " Jul 29 07:05:33.014409 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014106 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:33.014409 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014149 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:33.014814 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014780 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:05:33.014917 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.014821 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:33.015231 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.015150 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.015231 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.015178 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.015231 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.015194 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.016579 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.016556 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:05:33.016579 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.016560 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn" (OuterVolumeSpecName: "kube-api-access-mlsdn") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "kube-api-access-mlsdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:05:33.016784 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.016769 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:05:33.016865 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.016848 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c317822-e7d0-42c2-a3d2-68d5f5b55c40" (UID: "2c317822-e7d0-42c2-a3d2-68d5f5b55c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:33.116490 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116447 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.116490 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116487 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlsdn\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-kube-api-access-mlsdn\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.116490 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116499 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.116490 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116507 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.116747 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116515 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.116747 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.116522 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c317822-e7d0-42c2-a3d2-68d5f5b55c40-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:33.549325 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.549290 2556 generic.go:356] "Generic (PLEG): container finished" podID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerID="8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c" exitCode=0 Jul 29 07:05:33.549517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.549352 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" Jul 29 07:05:33.549517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.549370 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerDied","Data":"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c"} Jul 29 07:05:33.549517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.549439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt" event={"ID":"2c317822-e7d0-42c2-a3d2-68d5f5b55c40","Type":"ContainerDied","Data":"c60e5f2cfacc8617de1cf9067ed4a0b3a69ad45b45597bb7ad41a8db9780d2cd"} Jul 29 07:05:33.549517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.549464 2556 scope.go:117] "RemoveContainer" containerID="8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c" Jul 29 07:05:33.558273 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.558254 2556 scope.go:117] "RemoveContainer" containerID="6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459" Jul 29 07:05:33.565996 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.565980 2556 scope.go:117] "RemoveContainer" containerID="8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c" Jul 29 07:05:33.566247 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:05:33.566230 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c\": container with ID starting with 8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c not found: ID does not exist" containerID="8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c" Jul 29 07:05:33.566297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.566256 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c"} err="failed to get container status \"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c\": rpc error: code = NotFound desc = could not find container \"8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c\": container with ID starting with 8dfcaf57b6549c0354713b5e10940a919813a768b235174b2111321c8175207c not found: ID does not exist" Jul 29 07:05:33.566297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.566273 2556 scope.go:117] "RemoveContainer" containerID="6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459" Jul 29 07:05:33.566499 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:05:33.566480 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459\": container with ID starting with 6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459 not found: ID does not exist" containerID="6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459" Jul 29 07:05:33.566542 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.566509 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459"} err="failed to get container status \"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459\": rpc error: code = NotFound desc = could not find container \"6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459\": container with ID starting with 6eb2928fe24bda6ed4ceb0589fa69b08283fd131a352a917cef8f83c28d73459 not found: ID does not exist" Jul 29 07:05:33.570688 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.570665 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:05:33.574278 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.574255 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dd1393-openshift-default-9fc984c6b-c97mt"] Jul 29 07:05:33.927618 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:33.927527 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" path="/var/lib/kubelet/pods/2c317822-e7d0-42c2-a3d2-68d5f5b55c40/volumes" Jul 29 07:05:38.479584 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:38.479547 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:38.544091 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:38.544061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:38.604272 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:38.604239 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:38.655775 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:38.655744 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:39.013347 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:39.013319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:44.908265 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:44.908235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:44.959823 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:44.959789 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:45.151197 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.151164 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:05:45.151485 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.151460 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" containerID="cri-o://e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac" gracePeriod=30 Jul 29 07:05:45.174411 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.174316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:45.379436 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.379401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:45.739085 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.739055 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:45.995544 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995462 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:05:45.995894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995808 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:05:45.995894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995819 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:05:45.995894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995848 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8992f04b-e9cc-4235-816d-b594adb54f67" containerName="main" Jul 29 07:05:45.995894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995855 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8992f04b-e9cc-4235-816d-b594adb54f67" containerName="main" Jul 29 07:05:45.996024 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995929 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:05:45.996024 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.995947 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8992f04b-e9cc-4235-816d-b594adb54f67" containerName="main" Jul 29 07:05:45.998703 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:45.998683 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.001203 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.001180 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4b144e-openshift-default-dockercfg-287zb\"" Jul 29 07:05:46.010657 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.010632 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:05:46.044118 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.044085 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:46.127743 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127714 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.127906 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.127906 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.127906 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.128032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.128032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.128032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.127972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.128032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.128020 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.128150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.128044 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5v2\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228798 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228770 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5v2\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228857 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.228919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228908 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229161 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229161 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229161 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.228973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229319 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.229174 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229319 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.229286 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229463 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.229374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229463 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.229414 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.229934 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.229911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.231520 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.231498 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.231808 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.231786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.236749 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.236728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5v2\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.236843 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.236823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token\") pod \"e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.309239 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.309149 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:46.441397 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.441351 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:05:46.443126 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:05:46.443093 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb27d41_25c4_4623_a0a3_41bfbad1b1af.slice/crio-65d9b47f8f1f08df1ae7a62bc584b80d4dd157d1563fd2a943c2daa75f882096 WatchSource:0}: Error finding container 65d9b47f8f1f08df1ae7a62bc584b80d4dd157d1563fd2a943c2daa75f882096: Status 404 returned error can't find the container with id 65d9b47f8f1f08df1ae7a62bc584b80d4dd157d1563fd2a943c2daa75f882096 Jul 29 07:05:46.445208 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.445178 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:05:46.445289 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.445246 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:05:46.445289 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.445275 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:05:46.553905 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.553852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:46.608025 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.607942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerStarted","Data":"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787"} Jul 29 07:05:46.608025 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.608001 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerStarted","Data":"65d9b47f8f1f08df1ae7a62bc584b80d4dd157d1563fd2a943c2daa75f882096"} Jul 29 07:05:46.628173 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:46.628122 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podStartSLOduration=1.628103917 podStartE2EDuration="1.628103917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:05:46.626915291 +0000 UTC m=+2591.160994865" watchObservedRunningTime="2026-07-29 07:05:46.628103917 +0000 UTC m=+2591.162183489" Jul 29 07:05:47.309568 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:47.309529 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:47.314494 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:47.314471 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:47.612234 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:47.612138 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:47.613198 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:47.613182 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:05:48.503228 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:48.503190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:48.604320 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:48.604285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:48.742010 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:48.741975 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:48.880448 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:48.880360 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:49.196196 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:49.196161 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:49.293609 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:49.293570 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:05:50.404554 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.404528 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:05:50.574118 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574087 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574118 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574308 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574160 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574308 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574186 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574308 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574236 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574308 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574276 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lq9l\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574308 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574337 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574427 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket\") pod \"3bf0a136-6257-4c64-b389-e43bce21aa4f\" (UID: \"3bf0a136-6257-4c64-b389-e43bce21aa4f\") " Jul 29 07:05:50.574592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574426 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:50.574592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574550 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:50.574807 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574658 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:05:50.574807 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574745 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:50.575032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.574932 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.575100 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.575057 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.575100 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.575072 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.575100 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.575084 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf0a136-6257-4c64-b389-e43bce21aa4f-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.575421 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.575096 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:50.576512 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.576490 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:05:50.576872 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.576846 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:05:50.577076 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.577058 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l" (OuterVolumeSpecName: "kube-api-access-6lq9l") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "kube-api-access-6lq9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:05:50.577157 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.577084 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bf0a136-6257-4c64-b389-e43bce21aa4f" (UID: "3bf0a136-6257-4c64-b389-e43bce21aa4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:05:50.631811 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.631781 2556 generic.go:356] "Generic (PLEG): container finished" podID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerID="e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac" exitCode=0 Jul 29 07:05:50.631975 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.631881 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" Jul 29 07:05:50.631975 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.631877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerDied","Data":"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac"} Jul 29 07:05:50.632084 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.632031 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz" event={"ID":"3bf0a136-6257-4c64-b389-e43bce21aa4f","Type":"ContainerDied","Data":"00ddb362a12a159429807d15ed224350ff2624146e1d363724e1455d7c9aa2db"} Jul 29 07:05:50.632084 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.632053 2556 scope.go:117] "RemoveContainer" containerID="e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac" Jul 29 07:05:50.641641 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.641622 2556 scope.go:117] "RemoveContainer" containerID="ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb" Jul 29 07:05:50.649264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.649248 2556 scope.go:117] "RemoveContainer" containerID="e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac" Jul 29 07:05:50.649537 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:05:50.649518 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac\": container with ID starting with e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac not found: ID does not exist" containerID="e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac" Jul 29 07:05:50.649591 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.649545 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac"} err="failed to get container status \"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac\": rpc error: code = NotFound desc = could not find container \"e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac\": container with ID starting with e7d4debc7dbe860051dd211e8ea66f5942e75aec0f153d0a98dcd8cb2e9fdeac not found: ID does not exist" Jul 29 07:05:50.649591 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.649561 2556 scope.go:117] "RemoveContainer" containerID="ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb" Jul 29 07:05:50.649771 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:05:50.649752 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb\": container with ID starting with ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb not found: ID does not exist" containerID="ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb" Jul 29 07:05:50.649811 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.649777 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb"} err="failed to get container status \"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb\": rpc error: code = NotFound desc = could not find container \"ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb\": container with ID starting with ae957f153bff5a21310b22f9f7b942936e33facfe2181e6d8bd1c2b29b2022fb not found: ID does not exist" Jul 29 07:05:50.653567 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.653548 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:05:50.656521 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.656502 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9aa496-openshift-default-8468dbc5cb-xfpjz"] Jul 29 07:05:50.676261 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.676238 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.676261 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.676257 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lq9l\" (UniqueName: \"kubernetes.io/projected/3bf0a136-6257-4c64-b389-e43bce21aa4f-kube-api-access-6lq9l\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.676371 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.676272 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.676371 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.676285 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:50.676371 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:50.676293 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf0a136-6257-4c64-b389-e43bce21aa4f-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:05:51.927023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:05:51.926988 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" path="/var/lib/kubelet/pods/3bf0a136-6257-4c64-b389-e43bce21aa4f/volumes" Jul 29 07:06:02.614185 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.614142 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 29 07:06:02.614637 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.614204 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 29 07:06:02.695587 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.695559 2556 generic.go:356] "Generic (PLEG): container finished" podID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerID="2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787" exitCode=0 Jul 29 07:06:02.695745 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.695631 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerDied","Data":"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787"} Jul 29 07:06:02.696039 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.696025 2556 scope.go:117] "RemoveContainer" containerID="2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787" Jul 29 07:06:02.697053 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.697029 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:02.697121 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.697099 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:02.697164 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:02.697137 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:03.702324 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:03.702289 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerStarted","Data":"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5"} Jul 29 07:06:04.310291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:04.310256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:06:04.310624 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:04.310594 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 29 07:06:04.310766 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:04.310666 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 29 07:06:05.310436 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:05.310401 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 29 07:06:05.310818 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:05.310455 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 29 07:06:06.310510 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:06.310474 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 29 07:06:06.310924 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:06.310529 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 29 07:06:07.310120 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:07.310080 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 29 07:06:07.310321 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:07.310148 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 29 07:06:08.313571 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:08.313542 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:06:08.313939 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:08.313768 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:06:08.314345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:08.314327 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:06:13.472022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:13.471983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:13.934171 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:13.934091 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:14.216816 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.216778 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:06:14.217294 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217274 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:06:14.217345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217299 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:06:14.217345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217325 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.217345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217334 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.217476 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217358 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.217476 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217367 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.217547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217481 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.217547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217495 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c317822-e7d0-42c2-a3d2-68d5f5b55c40" containerName="istio-proxy" Jul 29 07:06:14.217547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.217507 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf0a136-6257-4c64-b389-e43bce21aa4f" containerName="istio-proxy" Jul 29 07:06:14.219459 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.219443 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.222233 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.222212 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e87d49-openshift-default-dockercfg-bp6mg\"" Jul 29 07:06:14.232183 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.232158 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:06:14.244895 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.244854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:14.269285 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269257 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269405 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269288 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269405 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269514 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269434 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269514 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269514 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269499 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269621 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269621 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269570 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6t9v\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.269621 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.269596 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.370899 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.370856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.370899 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.370903 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.370937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371044 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371090 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371129 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6t9v\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371254 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371314 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371622 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371528 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371786 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371764 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.371913 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.371896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.373199 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.373182 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.373697 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.373676 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.379182 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.379153 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.379325 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.379308 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6t9v\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v\") pod \"e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.530067 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.529975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:14.586455 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.586414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:14.660661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.660636 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:06:14.662541 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:06:14.662516 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deb4aff_4a87_4c00_9914_c14eb1a8ee6d.slice/crio-ab62a079b6e25f2729d1fda470100e0954747160064045bbb80a1f6838a2d10a WatchSource:0}: Error finding container ab62a079b6e25f2729d1fda470100e0954747160064045bbb80a1f6838a2d10a: Status 404 returned error can't find the container with id ab62a079b6e25f2729d1fda470100e0954747160064045bbb80a1f6838a2d10a Jul 29 07:06:14.665028 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.664995 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:14.665127 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.665072 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:14.665127 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.665115 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:14.752489 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.752445 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerStarted","Data":"6acaa7356cf39575ceb7c82ca9eb56a8ea240e74c6f3b2b882f2dc26bd985408"} Jul 29 07:06:14.752660 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.752509 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerStarted","Data":"ab62a079b6e25f2729d1fda470100e0954747160064045bbb80a1f6838a2d10a"} Jul 29 07:06:14.772028 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:14.771966 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podStartSLOduration=0.771949933 podStartE2EDuration="771.949933ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:06:14.770064064 +0000 UTC m=+2619.304143640" watchObservedRunningTime="2026-07-29 07:06:14.771949933 +0000 UTC m=+2619.306029505" Jul 29 07:06:15.530477 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:15.530439 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:15.535628 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:15.535602 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:15.758012 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:15.757983 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:15.758918 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:15.758895 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:16.657230 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:16.657201 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:16.756653 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:16.756616 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:16.857357 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:16.857324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:17.009404 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:17.009347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:17.310795 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:17.310722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:17.415718 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:17.415681 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:31.799546 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:31.799493 2556 patch_prober.go:28] interesting pod/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:06:31.800120 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:31.799568 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:06:32.842941 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.842910 2556 generic.go:356] "Generic (PLEG): container finished" podID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerID="6acaa7356cf39575ceb7c82ca9eb56a8ea240e74c6f3b2b882f2dc26bd985408" exitCode=0 Jul 29 07:06:32.843427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.842948 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerDied","Data":"6acaa7356cf39575ceb7c82ca9eb56a8ea240e74c6f3b2b882f2dc26bd985408"} Jul 29 07:06:32.843427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.843331 2556 scope.go:117] "RemoveContainer" containerID="6acaa7356cf39575ceb7c82ca9eb56a8ea240e74c6f3b2b882f2dc26bd985408" Jul 29 07:06:32.844336 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.844306 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:32.844474 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.844419 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:32.844474 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:32.844454 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:06:33.850605 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:33.850565 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerStarted","Data":"5664f5eac2697535d0f8c9a011e9620a4d52f1ce88b8a2fe622b8ea57bc43148"} Jul 29 07:06:34.531147 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:34.531111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:34.531346 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:34.531323 2556 patch_prober.go:28] interesting pod/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:06:34.531428 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:34.531396 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 07:06:35.531150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:35.531113 2556 patch_prober.go:28] interesting pod/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:06:35.531562 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:35.531181 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 07:06:36.530357 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:36.530319 2556 patch_prober.go:28] interesting pod/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:06:36.530550 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:36.530412 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 07:06:37.530672 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:37.530633 2556 patch_prober.go:28] interesting pod/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:06:37.531053 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:37.530691 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 07:06:38.534092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:38.534063 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:38.534616 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:38.534480 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:38.535066 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:38.535045 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:06:41.638577 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:41.638539 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:06:59.935797 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:06:59.935762 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:00.005012 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:00.004977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:00.077756 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:00.077724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:00.179394 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:00.179329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:00.466614 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:00.466565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:06.177543 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.177481 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:06.308178 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.308130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:06.453176 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.453147 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:07:06.453455 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.453427 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" containerID="cri-o://1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5" gracePeriod=30 Jul 29 07:07:06.535661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.535618 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:06.713323 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:06.713225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:08.316123 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:08.316082 2556 patch_prober.go:28] interesting pod/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:07:08.316518 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:08.316149 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:07:11.707747 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.707722 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:07:11.790479 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790414 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx5v2\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790479 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790476 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790652 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790567 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790652 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790615 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790652 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790641 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790810 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790666 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790810 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790711 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790810 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790741 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.790810 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790763 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs\") pod \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\" (UID: \"deb27d41-25c4-4623-a0a3-41bfbad1b1af\") " Jul 29 07:07:11.791012 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.790820 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:11.791068 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791012 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:11.791068 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791034 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:07:11.791174 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791145 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.791174 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791164 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.791174 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:11.791287 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791179 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.791849 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.791744 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data" (OuterVolumeSpecName: "istio-data") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:11.792821 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.792797 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2" (OuterVolumeSpecName: "kube-api-access-qx5v2") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "kube-api-access-qx5v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:07:11.792930 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.792843 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:07:11.793015 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.792994 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token" (OuterVolumeSpecName: "istio-token") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:07:11.793182 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.793168 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "deb27d41-25c4-4623-a0a3-41bfbad1b1af" (UID: "deb27d41-25c4-4623-a0a3-41bfbad1b1af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:11.892484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892449 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.892484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892482 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.892484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892491 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5v2\" (UniqueName: \"kubernetes.io/projected/deb27d41-25c4-4623-a0a3-41bfbad1b1af-kube-api-access-qx5v2\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.892651 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892500 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.892651 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892508 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:11.892651 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:11.892516 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb27d41-25c4-4623-a0a3-41bfbad1b1af-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:12.035814 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.035783 2556 generic.go:356] "Generic (PLEG): container finished" podID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerID="1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5" exitCode=0 Jul 29 07:07:12.035954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.035839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerDied","Data":"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5"} Jul 29 07:07:12.035954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.035846 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" Jul 29 07:07:12.035954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.035904 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d" event={"ID":"deb27d41-25c4-4623-a0a3-41bfbad1b1af","Type":"ContainerDied","Data":"65d9b47f8f1f08df1ae7a62bc584b80d4dd157d1563fd2a943c2daa75f882096"} Jul 29 07:07:12.035954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.035936 2556 scope.go:117] "RemoveContainer" containerID="1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5" Jul 29 07:07:12.044592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.044575 2556 scope.go:117] "RemoveContainer" containerID="2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787" Jul 29 07:07:12.052336 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.052317 2556 scope.go:117] "RemoveContainer" containerID="1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5" Jul 29 07:07:12.052637 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:07:12.052615 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5\": container with ID starting with 1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5 not found: ID does not exist" containerID="1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5" Jul 29 07:07:12.052699 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.052645 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5"} err="failed to get container status \"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5\": rpc error: code = NotFound desc = could not find container \"1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5\": container with ID starting with 1de12e0bd0637f468730de735881674c7a9150bde446eae9fad52db97b4be1e5 not found: ID does not exist" Jul 29 07:07:12.052699 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.052662 2556 scope.go:117] "RemoveContainer" containerID="2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787" Jul 29 07:07:12.052923 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:07:12.052908 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787\": container with ID starting with 2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787 not found: ID does not exist" containerID="2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787" Jul 29 07:07:12.052978 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.052927 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787"} err="failed to get container status \"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787\": rpc error: code = NotFound desc = could not find container \"2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787\": container with ID starting with 2de04b409addb0f2f92a3ecac89c2438053b9bcbe76ba42125bae97896eb0787 not found: ID does not exist" Jul 29 07:07:12.054044 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.054026 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:07:12.057928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:12.057909 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4b144e-openshift-default-6d96c5797b-bfz9d"] Jul 29 07:07:13.927198 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:13.927166 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" path="/var/lib/kubelet/pods/deb27d41-25c4-4623-a0a3-41bfbad1b1af/volumes" Jul 29 07:07:22.228911 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:22.228871 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:22.302395 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:22.302355 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:22.366908 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:22.366857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:22.469533 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:22.469503 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:22.772248 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:22.772209 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:28.647146 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:28.647017 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:28.748442 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:28.748409 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:28.942640 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:28.942598 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:07:28.942864 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:28.942840 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" containerID="cri-o://5664f5eac2697535d0f8c9a011e9620a4d52f1ce88b8a2fe622b8ea57bc43148" gracePeriod=30 Jul 29 07:07:28.969543 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:28.969511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:29.221949 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.221853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:29.592343 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.592251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:29.862325 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862232 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:07:29.862728 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862708 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:29.862728 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862722 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:29.862728 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862729 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:29.862872 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862734 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:29.862872 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.862793 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:29.866069 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.866053 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:29.868773 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.868735 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-50e693-openshift-default-dockercfg-8l2z6\"" Jul 29 07:07:29.875963 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.875942 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:07:29.905755 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:29.905727 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:30.043294 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.043261 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:30.058552 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058527 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058662 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058662 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058595 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058737 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpml\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058772 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058731 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058772 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058760 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058883 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058883 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058835 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.058883 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.058853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.159865 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.159775 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.159865 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.159816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.159865 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.159840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160128 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160176 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160294 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpml\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160348 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160348 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160336 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160546 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160520 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160617 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160582 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160617 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160604 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160723 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160678 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.160883 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.160866 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.162425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.162406 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.162669 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.162650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.169020 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.168990 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.169107 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.169072 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpml\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml\") pod \"e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.176995 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.176977 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:30.310621 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.310591 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:07:30.312570 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:07:30.312543 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6588ad46_c63f_4453_aea4_9ad36cfcf04e.slice/crio-36ef793cf3aef5013a2a3665a820593e16a5527dc7ac14a4c0c8e8b0e9afbc10 WatchSource:0}: Error finding container 36ef793cf3aef5013a2a3665a820593e16a5527dc7ac14a4c0c8e8b0e9afbc10: Status 404 returned error can't find the container with id 36ef793cf3aef5013a2a3665a820593e16a5527dc7ac14a4c0c8e8b0e9afbc10 Jul 29 07:07:30.314825 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.314787 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:30.314906 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.314872 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:30.314953 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:30.314913 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:31.118021 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:31.117985 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerStarted","Data":"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52"} Jul 29 07:07:31.118478 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:31.118060 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerStarted","Data":"36ef793cf3aef5013a2a3665a820593e16a5527dc7ac14a4c0c8e8b0e9afbc10"} Jul 29 07:07:31.138612 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:31.138554 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podStartSLOduration=2.13853628 podStartE2EDuration="2.13853628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:07:31.137043813 +0000 UTC m=+2695.671123385" watchObservedRunningTime="2026-07-29 07:07:31.13853628 +0000 UTC m=+2695.672615854" Jul 29 07:07:31.177886 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:31.177859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:31.183076 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:31.183055 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:32.122968 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:32.122931 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:32.123956 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:32.123938 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:33.030477 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.030445 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:33.123630 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.123601 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:33.365657 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.365558 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:33.640362 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.640277 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:33.743173 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.743136 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:33.856302 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:33.856275 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:34.134601 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.134566 2556 generic.go:356] "Generic (PLEG): container finished" podID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerID="5664f5eac2697535d0f8c9a011e9620a4d52f1ce88b8a2fe622b8ea57bc43148" exitCode=0 Jul 29 07:07:34.135042 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.134641 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerDied","Data":"5664f5eac2697535d0f8c9a011e9620a4d52f1ce88b8a2fe622b8ea57bc43148"} Jul 29 07:07:34.135042 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.134713 2556 scope.go:117] "RemoveContainer" containerID="6acaa7356cf39575ceb7c82ca9eb56a8ea240e74c6f3b2b882f2dc26bd985408" Jul 29 07:07:34.192681 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.192656 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:07:34.295794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295687 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.295794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295727 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.295794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295763 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.295794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295779 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6t9v\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.295794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295800 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295816 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295847 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295873 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.295907 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket\") pod \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\" (UID: \"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d\") " Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296160 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:34.296226 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296224 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.296547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296373 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:34.296547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296531 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:07:34.296728 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296703 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:34.296728 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.296717 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:34.298500 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.298352 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v" (OuterVolumeSpecName: "kube-api-access-q6t9v") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "kube-api-access-q6t9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:07:34.298616 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.298550 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:07:34.298616 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.298602 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:07:34.298709 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.298652 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" (UID: "2deb4aff-4a87-4c00-9914-c14eb1a8ee6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:07:34.397449 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397420 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397449 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397447 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397449 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397456 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397465 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6t9v\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-kube-api-access-q6t9v\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397474 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397482 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397490 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:34.397655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:34.397497 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:07:35.140751 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.140710 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" event={"ID":"2deb4aff-4a87-4c00-9914-c14eb1a8ee6d","Type":"ContainerDied","Data":"ab62a079b6e25f2729d1fda470100e0954747160064045bbb80a1f6838a2d10a"} Jul 29 07:07:35.141190 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.140778 2556 scope.go:117] "RemoveContainer" containerID="5664f5eac2697535d0f8c9a011e9620a4d52f1ce88b8a2fe622b8ea57bc43148" Jul 29 07:07:35.141190 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.140731 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd" Jul 29 07:07:35.162741 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.162715 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:07:35.168396 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.168356 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e87d49-openshift-default-f94fddcd8-8qgxd"] Jul 29 07:07:35.929951 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:35.929920 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" path="/var/lib/kubelet/pods/2deb4aff-4a87-4c00-9914-c14eb1a8ee6d/volumes" Jul 29 07:07:47.124932 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.124888 2556 patch_prober.go:28] interesting pod/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 29 07:07:47.125341 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.124943 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 29 07:07:47.201203 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.201168 2556 generic.go:356] "Generic (PLEG): container finished" podID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerID="dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52" exitCode=0 Jul 29 07:07:47.201413 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.201248 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerDied","Data":"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52"} Jul 29 07:07:47.201725 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.201708 2556 scope.go:117] "RemoveContainer" containerID="dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52" Jul 29 07:07:47.202750 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.202718 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:47.202840 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.202787 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:47.202840 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:47.202816 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:48.207598 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:48.207562 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerStarted","Data":"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe"} Jul 29 07:07:49.177197 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:49.177152 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:49.177499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:49.177472 2556 patch_prober.go:28] interesting pod/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 29 07:07:49.177601 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:49.177531 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 29 07:07:50.178223 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:50.178178 2556 patch_prober.go:28] interesting pod/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 29 07:07:50.178645 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:50.178264 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 29 07:07:51.177955 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:51.177915 2556 patch_prober.go:28] interesting pod/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 29 07:07:51.178136 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:51.178004 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 29 07:07:52.178304 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:52.178263 2556 patch_prober.go:28] interesting pod/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 29 07:07:52.178694 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:52.178319 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 29 07:07:53.181115 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:53.181081 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:53.181499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:53.181343 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:53.182047 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:53.182030 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:07:58.106054 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.106022 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:58.487088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.487044 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:58.745368 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745295 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:07:58.745696 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745680 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.745740 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745701 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.745740 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745715 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.745740 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745720 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.745836 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745773 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb27d41-25c4-4623-a0a3-41bfbad1b1af" containerName="istio-proxy" Jul 29 07:07:58.745836 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745785 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.745836 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.745792 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deb4aff-4a87-4c00-9914-c14eb1a8ee6d" containerName="istio-proxy" Jul 29 07:07:58.747966 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.747946 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.750934 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.750917 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1e6a9b-openshift-default-dockercfg-t987x\"" Jul 29 07:07:58.763197 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.763167 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:07:58.774525 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.774504 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:07:58.832584 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832555 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832703 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832703 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832709 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832769 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cczh\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832790 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.832928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.832893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.933624 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.933802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cczh\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.933802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.933802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.933802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933801 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933844 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.933910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934224 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.934143 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934224 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.934187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.934240 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.934262 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.934426 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.934410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.936239 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.936218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.936404 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.936284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.942263 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.942239 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cczh\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:58.942554 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:58.942526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token\") pod \"e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:59.057612 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.057544 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:07:59.189678 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.189644 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:07:59.190829 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:07:59.190806 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b68fe7_3036_435c_89ab_58ad7e3c25c8.slice/crio-c85fe54f30b1f3e1a8d861eda70d7c1e30426c20669032123e48d4b7d1d96eb7 WatchSource:0}: Error finding container c85fe54f30b1f3e1a8d861eda70d7c1e30426c20669032123e48d4b7d1d96eb7: Status 404 returned error can't find the container with id c85fe54f30b1f3e1a8d861eda70d7c1e30426c20669032123e48d4b7d1d96eb7 Jul 29 07:07:59.193088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.193052 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:59.193159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.193136 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:59.193198 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.193175 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:07:59.259784 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.259758 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerStarted","Data":"c85fe54f30b1f3e1a8d861eda70d7c1e30426c20669032123e48d4b7d1d96eb7"} Jul 29 07:07:59.541423 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:07:59.541390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:00.265446 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:00.265412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerStarted","Data":"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594"} Jul 29 07:08:00.289051 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:00.289004 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podStartSLOduration=2.288989339 podStartE2EDuration="2.288989339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:08:00.287361486 +0000 UTC m=+2724.821441059" watchObservedRunningTime="2026-07-29 07:08:00.288989339 +0000 UTC m=+2724.823068912" Jul 29 07:08:01.058253 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:01.058224 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:01.063213 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:01.063187 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:01.269931 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:01.269899 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:01.270932 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:01.270910 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:02.435095 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:02.435027 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:02.548002 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:02.547954 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:02.805716 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:02.805683 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:03.530027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:03.529991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:03.654164 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:03.654134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:17.311559 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:17.311517 2556 patch_prober.go:28] interesting pod/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:08:17.312051 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:17.311578 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:08:18.344539 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.344510 2556 generic.go:356] "Generic (PLEG): container finished" podID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerID="ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594" exitCode=0 Jul 29 07:08:18.344973 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.344584 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerDied","Data":"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594"} Jul 29 07:08:18.345097 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.345077 2556 scope.go:117] "RemoveContainer" containerID="ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594" Jul 29 07:08:18.346087 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.346065 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:08:18.346140 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.346130 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:08:18.346177 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:18.346160 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:08:19.352420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:19.352357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerStarted","Data":"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817"} Jul 29 07:08:20.057818 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:20.057778 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:20.058026 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:20.058005 2556 patch_prober.go:28] interesting pod/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 29 07:08:20.058108 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:20.058064 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 29 07:08:21.058801 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:21.058762 2556 patch_prober.go:28] interesting pod/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 29 07:08:21.059814 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:21.059778 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 29 07:08:22.057964 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:22.057926 2556 patch_prober.go:28] interesting pod/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 29 07:08:22.058292 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:22.058251 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 29 07:08:23.058548 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:23.058509 2556 patch_prober.go:28] interesting pod/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Jul 29 07:08:23.059098 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:23.058579 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Jul 29 07:08:24.061172 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:24.061134 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:24.061589 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:24.061409 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:24.061971 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:24.061954 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:08:27.335425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:27.335372 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:33.748592 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.748549 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:08:33.751112 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.751088 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.754885 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.754861 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-50e693\"/\"kube-root-ca.crt\"" Jul 29 07:08:33.756081 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.756048 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-50e693\"/\"default-dockercfg-5z74d\"" Jul 29 07:08:33.756321 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.756279 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-50e693\"/\"auth-test-model-50e693-kserve-self-signed-certs\"" Jul 29 07:08:33.756321 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.756295 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-50e693\"/\"openshift-service-ca.crt\"" Jul 29 07:08:33.772641 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.772612 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:08:33.785512 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.785473 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:08:33.858257 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858223 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.858486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.858486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858323 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4r9v\" (UniqueName: \"kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.858486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.858486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.858486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.858481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959363 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959596 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959410 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959596 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959596 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959780 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959780 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959655 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4r9v\" (UniqueName: \"kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959900 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959794 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959900 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.959985 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.959963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.961791 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.961772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.962167 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.962146 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:33.971789 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:33.971764 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4r9v\" (UniqueName: \"kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v\") pod \"auth-test-model-50e693-kserve-69775856cf-r9ph5\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:34.060896 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:34.060810 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:34.205213 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:34.205189 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:08:34.207276 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:08:34.207248 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d98dc5a_6143_4474_b24c_ab42f19d5022.slice/crio-906d01d9ffde48d1f4810ca4aea46b4ccfc24719d970d9716809342526436e4b WatchSource:0}: Error finding container 906d01d9ffde48d1f4810ca4aea46b4ccfc24719d970d9716809342526436e4b: Status 404 returned error can't find the container with id 906d01d9ffde48d1f4810ca4aea46b4ccfc24719d970d9716809342526436e4b Jul 29 07:08:34.426835 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:34.426747 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" event={"ID":"1d98dc5a-6143-4474-b24c-ab42f19d5022","Type":"ContainerStarted","Data":"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91"} Jul 29 07:08:34.426835 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:34.426786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" event={"ID":"1d98dc5a-6143-4474-b24c-ab42f19d5022","Type":"ContainerStarted","Data":"906d01d9ffde48d1f4810ca4aea46b4ccfc24719d970d9716809342526436e4b"} Jul 29 07:08:34.459393 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:34.459337 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" podStartSLOduration=1.459321977 podStartE2EDuration="1.459321977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:08:34.458786832 +0000 UTC m=+2758.992866405" watchObservedRunningTime="2026-07-29 07:08:34.459321977 +0000 UTC m=+2758.993401550" Jul 29 07:08:44.061840 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:44.061808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:44.061840 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:44.061857 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:44.074623 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:44.074591 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:08:44.482080 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:08:44.482052 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:09:02.083289 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:09:02.083201 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:09:11.525778 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:09:11.525728 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:09:13.393173 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:09:13.393132 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:09:13.493981 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:09:13.493949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:10:18.163683 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:10:18.163642 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:11:01.837476 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:11:01.837395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:11:52.005608 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:11:52.005570 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:13:44.090517 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:44.090439 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:13:44.143159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:44.143116 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:13:44.959695 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:44.959659 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:13:59.109796 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.109755 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" podUID="1d98dc5a-6143-4474-b24c-ab42f19d5022" containerName="main" containerID="cri-o://9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91" gracePeriod=45 Jul 29 07:13:59.367470 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.367393 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:13:59.465085 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465052 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465110 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465137 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465250 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465202 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465451 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465257 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4r9v\" (UniqueName: \"kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465451 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465309 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir\") pod \"1d98dc5a-6143-4474-b24c-ab42f19d5022\" (UID: \"1d98dc5a-6143-4474-b24c-ab42f19d5022\") " Jul 29 07:13:59.465569 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465511 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:13:59.465619 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465564 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home" (OuterVolumeSpecName: "home") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:13:59.465619 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465605 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:13:59.465732 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465668 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.465732 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.465690 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.467522 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.467499 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:13:59.467631 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.467529 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm" (OuterVolumeSpecName: "dshm") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:13:59.467722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.467705 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v" (OuterVolumeSpecName: "kube-api-access-g4r9v") pod "1d98dc5a-6143-4474-b24c-ab42f19d5022" (UID: "1d98dc5a-6143-4474-b24c-ab42f19d5022"). InnerVolumeSpecName "kube-api-access-g4r9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:13:59.566334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.566290 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.566334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.566327 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d98dc5a-6143-4474-b24c-ab42f19d5022-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.566334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.566337 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d98dc5a-6143-4474-b24c-ab42f19d5022-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.566334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.566347 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4r9v\" (UniqueName: \"kubernetes.io/projected/1d98dc5a-6143-4474-b24c-ab42f19d5022-kube-api-access-g4r9v\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:13:59.851864 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.851827 2556 generic.go:356] "Generic (PLEG): container finished" podID="1d98dc5a-6143-4474-b24c-ab42f19d5022" containerID="9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91" exitCode=0 Jul 29 07:13:59.852079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.851873 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" Jul 29 07:13:59.852079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.851916 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" event={"ID":"1d98dc5a-6143-4474-b24c-ab42f19d5022","Type":"ContainerDied","Data":"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91"} Jul 29 07:13:59.852079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.851967 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5" event={"ID":"1d98dc5a-6143-4474-b24c-ab42f19d5022","Type":"ContainerDied","Data":"906d01d9ffde48d1f4810ca4aea46b4ccfc24719d970d9716809342526436e4b"} Jul 29 07:13:59.852079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.851990 2556 scope.go:117] "RemoveContainer" containerID="9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91" Jul 29 07:13:59.860809 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.860791 2556 scope.go:117] "RemoveContainer" containerID="9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91" Jul 29 07:13:59.861085 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:13:59.861061 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91\": container with ID starting with 9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91 not found: ID does not exist" containerID="9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91" Jul 29 07:13:59.861160 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.861096 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91"} err="failed to get container status \"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91\": rpc error: code = NotFound desc = could not find container \"9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91\": container with ID starting with 9a75e9d97fe75f9557bb58904d4d392c1ce50ee26c97eb79af1c29d86fdbdb91 not found: ID does not exist" Jul 29 07:13:59.871707 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.871688 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:13:59.873896 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.873876 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-50e693/auth-test-model-50e693-kserve-69775856cf-r9ph5"] Jul 29 07:13:59.927182 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:13:59.927148 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d98dc5a-6143-4474-b24c-ab42f19d5022" path="/var/lib/kubelet/pods/1d98dc5a-6143-4474-b24c-ab42f19d5022/volumes" Jul 29 07:14:01.439300 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:01.439260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:14:24.214368 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.214334 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:14:24.214900 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.214879 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d98dc5a-6143-4474-b24c-ab42f19d5022" containerName="main" Jul 29 07:14:24.214975 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.214904 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d98dc5a-6143-4474-b24c-ab42f19d5022" containerName="main" Jul 29 07:14:24.215027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.214996 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d98dc5a-6143-4474-b24c-ab42f19d5022" containerName="main" Jul 29 07:14:24.216806 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.216786 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.221508 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.221485 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"sub-test-model-1e6a9b-kserve-self-signed-certs\"" Jul 29 07:14:24.221619 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.221488 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"default-dockercfg-xw6xs\"" Jul 29 07:14:24.221619 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.221525 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"kube-root-ca.crt\"" Jul 29 07:14:24.221619 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.221492 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"openshift-service-ca.crt\"" Jul 29 07:14:24.229260 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.229235 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:14:24.240022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.239987 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:14:24.277027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.276990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.277027 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.277028 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.277248 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.277089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.277248 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.277156 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvb4s\" (UniqueName: \"kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.277248 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.277218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.277248 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.277238 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.378799 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.378799 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378903 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.378935 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvb4s\" (UniqueName: \"kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.379145 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379677 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.379648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.379833 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.379687 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.381809 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.381786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.382326 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.382305 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.388132 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.388109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvb4s\" (UniqueName: \"kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s\") pod \"sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.526124 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.526030 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:24.662224 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.662197 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:14:24.664318 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:14:24.664293 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c3c479d_840c_43f1_86a2_c35b97ccd156.slice/crio-f29ffa9640f255e88bf89b3a1fe98699d01572d5361869df92bc787408295d72 WatchSource:0}: Error finding container f29ffa9640f255e88bf89b3a1fe98699d01572d5361869df92bc787408295d72: Status 404 returned error can't find the container with id f29ffa9640f255e88bf89b3a1fe98699d01572d5361869df92bc787408295d72 Jul 29 07:14:24.962165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.962129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" event={"ID":"8c3c479d-840c-43f1-86a2-c35b97ccd156","Type":"ContainerStarted","Data":"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce"} Jul 29 07:14:24.962334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.962174 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" event={"ID":"8c3c479d-840c-43f1-86a2-c35b97ccd156","Type":"ContainerStarted","Data":"f29ffa9640f255e88bf89b3a1fe98699d01572d5361869df92bc787408295d72"} Jul 29 07:14:24.981502 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:24.981444 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" podStartSLOduration=0.981426957 podStartE2EDuration="981.426957ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:14:24.979886994 +0000 UTC m=+3109.513966568" watchObservedRunningTime="2026-07-29 07:14:24.981426957 +0000 UTC m=+3109.515506540" Jul 29 07:14:34.526758 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:34.526719 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:34.527263 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:34.526772 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:34.540029 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:34.540002 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:14:35.018736 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:14:35.018707 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:15:40.580349 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:40.580310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:44.135484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:44.135441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:45.108539 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:45.108504 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:15:45.121817 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:45.121776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:53.175128 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:53.175088 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:15:59.590462 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:15:59.590423 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:00.127720 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.127678 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" podUID="8c3c479d-840c-43f1-86a2-c35b97ccd156" containerName="main" containerID="cri-o://2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce" gracePeriod=45 Jul 29 07:16:00.367107 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.367085 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:16:00.389057 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.388987 2556 generic.go:356] "Generic (PLEG): container finished" podID="8c3c479d-840c-43f1-86a2-c35b97ccd156" containerID="2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce" exitCode=0 Jul 29 07:16:00.389057 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.389024 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" Jul 29 07:16:00.389191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.389074 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" event={"ID":"8c3c479d-840c-43f1-86a2-c35b97ccd156","Type":"ContainerDied","Data":"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce"} Jul 29 07:16:00.389191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.389118 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz" event={"ID":"8c3c479d-840c-43f1-86a2-c35b97ccd156","Type":"ContainerDied","Data":"f29ffa9640f255e88bf89b3a1fe98699d01572d5361869df92bc787408295d72"} Jul 29 07:16:00.389191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.389138 2556 scope.go:117] "RemoveContainer" containerID="2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce" Jul 29 07:16:00.397685 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.397660 2556 scope.go:117] "RemoveContainer" containerID="2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce" Jul 29 07:16:00.398250 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:16:00.398224 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce\": container with ID starting with 2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce not found: ID does not exist" containerID="2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce" Jul 29 07:16:00.398411 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.398258 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce"} err="failed to get container status \"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce\": rpc error: code = NotFound desc = could not find container \"2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce\": container with ID starting with 2c250949cd599a9acf0756e20fa795ae490bcaaca35a4fc6ed1a630ea1e846ce not found: ID does not exist" Jul 29 07:16:00.464794 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.464765 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.464964 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.464806 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.464964 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.464866 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvb4s\" (UniqueName: \"kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.464964 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.464919 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.465139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.464983 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.465139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465017 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs\") pod \"8c3c479d-840c-43f1-86a2-c35b97ccd156\" (UID: \"8c3c479d-840c-43f1-86a2-c35b97ccd156\") " Jul 29 07:16:00.465291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465155 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home" (OuterVolumeSpecName: "home") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:16:00.465291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465249 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:16:00.465291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465256 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:16:00.465473 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465421 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.465473 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465435 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.465473 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.465444 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.467243 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.467220 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm" (OuterVolumeSpecName: "dshm") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:16:00.467345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.467296 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s" (OuterVolumeSpecName: "kube-api-access-kvb4s") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "kube-api-access-kvb4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:16:00.467345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.467294 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c3c479d-840c-43f1-86a2-c35b97ccd156" (UID: "8c3c479d-840c-43f1-86a2-c35b97ccd156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:16:00.566551 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.566516 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3c479d-840c-43f1-86a2-c35b97ccd156-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.566551 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.566544 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3c479d-840c-43f1-86a2-c35b97ccd156-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.566551 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.566554 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvb4s\" (UniqueName: \"kubernetes.io/projected/8c3c479d-840c-43f1-86a2-c35b97ccd156-kube-api-access-kvb4s\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:16:00.714288 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.714260 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:16:00.718924 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:00.718895 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/sub-test-model-1e6a9b-kserve-65bcfdf98c-nd2xz"] Jul 29 07:16:01.927121 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:01.927088 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3c479d-840c-43f1-86a2-c35b97ccd156" path="/var/lib/kubelet/pods/8c3c479d-840c-43f1-86a2-c35b97ccd156/volumes" Jul 29 07:16:17.610238 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.610209 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:16:17.610678 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.610567 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3c479d-840c-43f1-86a2-c35b97ccd156" containerName="main" Jul 29 07:16:17.610678 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.610579 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3c479d-840c-43f1-86a2-c35b97ccd156" containerName="main" Jul 29 07:16:17.610678 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.610645 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3c479d-840c-43f1-86a2-c35b97ccd156" containerName="main" Jul 29 07:16:17.613416 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.613399 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.617724 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.617696 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"kube-root-ca.crt\"" Jul 29 07:16:17.617874 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.617760 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"rate-test-model-1e6a9b-kserve-self-signed-certs\"" Jul 29 07:16:17.617874 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.617757 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"default-dockercfg-xw6xs\"" Jul 29 07:16:17.617874 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.617757 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e6a9b\"/\"openshift-service-ca.crt\"" Jul 29 07:16:17.624077 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.624054 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:16:17.638887 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.638857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:16:17.724919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.724888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.724919 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.724920 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.725127 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.724945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.725127 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.725080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.725200 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.725141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.725200 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.725174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnqf\" (UniqueName: \"kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826739 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826514 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826739 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826562 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826739 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826599 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826739 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnqf\" (UniqueName: \"kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.826889 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.827014 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.827014 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.826989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.828957 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.828929 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.829124 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.829105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.836109 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.836087 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnqf\" (UniqueName: \"kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf\") pod \"rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:17.922241 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:17.922165 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:18.059989 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:18.059958 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:16:18.060997 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:16:18.060955 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e10614_c0e2_4269_8673_caa85810ae52.slice/crio-2d9c6e78799cdeeda1b55c33d2064defe1c895dd004236e3c4a89165229dc27b WatchSource:0}: Error finding container 2d9c6e78799cdeeda1b55c33d2064defe1c895dd004236e3c4a89165229dc27b: Status 404 returned error can't find the container with id 2d9c6e78799cdeeda1b55c33d2064defe1c895dd004236e3c4a89165229dc27b Jul 29 07:16:18.474843 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:18.474809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" event={"ID":"36e10614-c0e2-4269-8673-caa85810ae52","Type":"ContainerStarted","Data":"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096"} Jul 29 07:16:18.474843 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:18.474850 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" event={"ID":"36e10614-c0e2-4269-8673-caa85810ae52","Type":"ContainerStarted","Data":"2d9c6e78799cdeeda1b55c33d2064defe1c895dd004236e3c4a89165229dc27b"} Jul 29 07:16:18.496006 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:18.495952 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" podStartSLOduration=1.495935522 podStartE2EDuration="1.495935522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:18.495367744 +0000 UTC m=+3223.029447316" watchObservedRunningTime="2026-07-29 07:16:18.495935522 +0000 UTC m=+3223.030015126" Jul 29 07:16:27.922493 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:27.922394 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:27.922493 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:27.922459 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:27.935156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:27.935125 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:16:28.532298 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:16:28.532263 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:17:45.131757 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:45.131724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:45.396503 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:45.396426 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:17:45.415297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:45.415265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:17:52.694441 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.694405 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:52.697650 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.697633 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:52.700689 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.700657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8cb9r\"" Jul 29 07:17:52.702057 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.702034 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:52.864791 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.864741 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kmv\" (UniqueName: \"kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv\") pod \"test-curl-7822-140659762878480\" (UID: \"fcd3a1e1-5166-44c6-89ef-0f068b989621\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:52.966225 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.966134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kmv\" (UniqueName: \"kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv\") pod \"test-curl-7822-140659762878480\" (UID: \"fcd3a1e1-5166-44c6-89ef-0f068b989621\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:52.975008 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:52.974977 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kmv\" (UniqueName: \"kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv\") pod \"test-curl-7822-140659762878480\" (UID: \"fcd3a1e1-5166-44c6-89ef-0f068b989621\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:53.008491 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.008467 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:53.150549 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.150521 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:53.152743 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:17:53.152718 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcd3a1e1_5166_44c6_89ef_0f068b989621.slice/crio-78b18598595bebb0ebae23ee92e4a5e4911aaffa66c2cf465ca6b901458e6bc1 WatchSource:0}: Error finding container 78b18598595bebb0ebae23ee92e4a5e4911aaffa66c2cf465ca6b901458e6bc1: Status 404 returned error can't find the container with id 78b18598595bebb0ebae23ee92e4a5e4911aaffa66c2cf465ca6b901458e6bc1 Jul 29 07:17:53.154478 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.154448 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:17:53.906305 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.906276 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762878480" event={"ID":"fcd3a1e1-5166-44c6-89ef-0f068b989621","Type":"ContainerStarted","Data":"1abb8b95b22214de097627ae5cfb0779514e27a708ed2d2530e5a53191e8bc44"} Jul 29 07:17:53.906644 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.906318 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762878480" event={"ID":"fcd3a1e1-5166-44c6-89ef-0f068b989621","Type":"ContainerStarted","Data":"78b18598595bebb0ebae23ee92e4a5e4911aaffa66c2cf465ca6b901458e6bc1"} Jul 29 07:17:53.922331 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:53.922289 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7822-140659762878480" podStartSLOduration=1.251078173 podStartE2EDuration="1.92227271s" totalImagesPullingTime="671.194537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:17:53.921637815 +0000 UTC m=+3318.455717389" watchObservedRunningTime="2026-07-29 07:17:53.92227271 +0000 UTC m=+3318.456352305" Jul 29 07:17:54.011991 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:54.011956 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659762878480_fcd3a1e1-5166-44c6-89ef-0f068b989621/test-curl-7822-140659762878480/0.log" Jul 29 07:17:54.912486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:54.912452 2556 generic.go:356] "Generic (PLEG): container finished" podID="fcd3a1e1-5166-44c6-89ef-0f068b989621" containerID="1abb8b95b22214de097627ae5cfb0779514e27a708ed2d2530e5a53191e8bc44" exitCode=0 Jul 29 07:17:54.912862 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:54.912547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762878480" event={"ID":"fcd3a1e1-5166-44c6-89ef-0f068b989621","Type":"ContainerDied","Data":"1abb8b95b22214de097627ae5cfb0779514e27a708ed2d2530e5a53191e8bc44"} Jul 29 07:17:56.054197 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.054176 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.086524 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.086497 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:56.091368 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.091332 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87kmv\" (UniqueName: \"kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv\") pod \"fcd3a1e1-5166-44c6-89ef-0f068b989621\" (UID: \"fcd3a1e1-5166-44c6-89ef-0f068b989621\") " Jul 29 07:17:56.092079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.092059 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:56.093632 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.093603 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv" (OuterVolumeSpecName: "kube-api-access-87kmv") pod "fcd3a1e1-5166-44c6-89ef-0f068b989621" (UID: "fcd3a1e1-5166-44c6-89ef-0f068b989621"). InnerVolumeSpecName "kube-api-access-87kmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:17:56.192947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.192919 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kmv\" (UniqueName: \"kubernetes.io/projected/fcd3a1e1-5166-44c6-89ef-0f068b989621-kube-api-access-87kmv\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:17:56.212979 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.212951 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:56.213271 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.213259 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd3a1e1-5166-44c6-89ef-0f068b989621" containerName="test-curl-7822-140659762878480" Jul 29 07:17:56.213310 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.213273 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd3a1e1-5166-44c6-89ef-0f068b989621" containerName="test-curl-7822-140659762878480" Jul 29 07:17:56.213345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.213326 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd3a1e1-5166-44c6-89ef-0f068b989621" containerName="test-curl-7822-140659762878480" Jul 29 07:17:56.216217 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.216202 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.220703 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.220676 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:56.293615 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.293544 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47n4x\" (UniqueName: \"kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x\") pod \"test-curl-7822-140659762878480\" (UID: \"c35dd254-2a35-4561-9e38-1aa02d74b874\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.394882 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.394845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47n4x\" (UniqueName: \"kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x\") pod \"test-curl-7822-140659762878480\" (UID: \"c35dd254-2a35-4561-9e38-1aa02d74b874\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.414415 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.414395 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47n4x\" (UniqueName: \"kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x\") pod \"test-curl-7822-140659762878480\" (UID: \"c35dd254-2a35-4561-9e38-1aa02d74b874\") " pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.526149 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.526080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.652163 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.652134 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:56.653680 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:17:56.653653 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc35dd254_2a35_4561_9e38_1aa02d74b874.slice/crio-e76c9a8042990687ef606af12b1d0b5be767080816ef2725b1bc93c7b36af293 WatchSource:0}: Error finding container e76c9a8042990687ef606af12b1d0b5be767080816ef2725b1bc93c7b36af293: Status 404 returned error can't find the container with id e76c9a8042990687ef606af12b1d0b5be767080816ef2725b1bc93c7b36af293 Jul 29 07:17:56.922521 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.922437 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:56.922521 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.922464 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b18598595bebb0ebae23ee92e4a5e4911aaffa66c2cf465ca6b901458e6bc1" Jul 29 07:17:56.923499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.923478 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762878480" event={"ID":"c35dd254-2a35-4561-9e38-1aa02d74b874","Type":"ContainerStarted","Data":"e76c9a8042990687ef606af12b1d0b5be767080816ef2725b1bc93c7b36af293"} Jul 29 07:17:56.934159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:56.934130 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7822-140659762878480" oldPodUID="fcd3a1e1-5166-44c6-89ef-0f068b989621" podUID="c35dd254-2a35-4561-9e38-1aa02d74b874" Jul 29 07:17:57.928060 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:57.928016 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd3a1e1-5166-44c6-89ef-0f068b989621" path="/var/lib/kubelet/pods/fcd3a1e1-5166-44c6-89ef-0f068b989621/volumes" Jul 29 07:17:57.930124 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:57.930094 2556 generic.go:356] "Generic (PLEG): container finished" podID="c35dd254-2a35-4561-9e38-1aa02d74b874" containerID="1ec0c2dce368b51c8bb07465f09df6783b5dea3d9880dc3fd32c265c26ba524e" exitCode=0 Jul 29 07:17:57.930273 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:57.930147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762878480" event={"ID":"c35dd254-2a35-4561-9e38-1aa02d74b874","Type":"ContainerDied","Data":"1ec0c2dce368b51c8bb07465f09df6783b5dea3d9880dc3fd32c265c26ba524e"} Jul 29 07:17:59.067906 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.067880 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:59.088167 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.088137 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659762878480_c35dd254-2a35-4561-9e38-1aa02d74b874/test-curl-7822-140659762878480/0.log" Jul 29 07:17:59.113327 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.113297 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:59.118570 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.118548 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659762878480"] Jul 29 07:17:59.120184 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.120166 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47n4x\" (UniqueName: \"kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x\") pod \"c35dd254-2a35-4561-9e38-1aa02d74b874\" (UID: \"c35dd254-2a35-4561-9e38-1aa02d74b874\") " Jul 29 07:17:59.122338 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.122311 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x" (OuterVolumeSpecName: "kube-api-access-47n4x") pod "c35dd254-2a35-4561-9e38-1aa02d74b874" (UID: "c35dd254-2a35-4561-9e38-1aa02d74b874"). InnerVolumeSpecName "kube-api-access-47n4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:17:59.221827 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.221791 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47n4x\" (UniqueName: \"kubernetes.io/projected/c35dd254-2a35-4561-9e38-1aa02d74b874-kube-api-access-47n4x\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:17:59.235445 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.235418 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:17:59.235747 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.235735 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35dd254-2a35-4561-9e38-1aa02d74b874" containerName="test-curl-7822-140659762878480" Jul 29 07:17:59.235790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.235749 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35dd254-2a35-4561-9e38-1aa02d74b874" containerName="test-curl-7822-140659762878480" Jul 29 07:17:59.235825 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.235811 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35dd254-2a35-4561-9e38-1aa02d74b874" containerName="test-curl-7822-140659762878480" Jul 29 07:17:59.238799 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.238777 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:17:59.245126 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.245101 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:17:59.322485 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.322445 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltst\" (UniqueName: \"kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst\") pod \"test-curl-7822-140659762879488\" (UID: \"44b0be51-f85a-4a17-910a-67dd831efa9d\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:17:59.423286 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.423255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltst\" (UniqueName: \"kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst\") pod \"test-curl-7822-140659762879488\" (UID: \"44b0be51-f85a-4a17-910a-67dd831efa9d\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:17:59.432152 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.432126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltst\" (UniqueName: \"kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst\") pod \"test-curl-7822-140659762879488\" (UID: \"44b0be51-f85a-4a17-910a-67dd831efa9d\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:17:59.549403 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.549286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:17:59.677269 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.677246 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:17:59.679361 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:17:59.679336 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b0be51_f85a_4a17_910a_67dd831efa9d.slice/crio-4de65c6e16936c0580d258580fa2643da9b4c466d2f0839263295456e58f4c1a WatchSource:0}: Error finding container 4de65c6e16936c0580d258580fa2643da9b4c466d2f0839263295456e58f4c1a: Status 404 returned error can't find the container with id 4de65c6e16936c0580d258580fa2643da9b4c466d2f0839263295456e58f4c1a Jul 29 07:17:59.928139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.928058 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35dd254-2a35-4561-9e38-1aa02d74b874" path="/var/lib/kubelet/pods/c35dd254-2a35-4561-9e38-1aa02d74b874/volumes" Jul 29 07:17:59.939664 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.939643 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762878480" Jul 29 07:17:59.939793 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.939644 2556 scope.go:117] "RemoveContainer" containerID="1ec0c2dce368b51c8bb07465f09df6783b5dea3d9880dc3fd32c265c26ba524e" Jul 29 07:17:59.940969 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:17:59.940879 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762879488" event={"ID":"44b0be51-f85a-4a17-910a-67dd831efa9d","Type":"ContainerStarted","Data":"4de65c6e16936c0580d258580fa2643da9b4c466d2f0839263295456e58f4c1a"} Jul 29 07:18:00.415249 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.415207 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" podUID="36e10614-c0e2-4269-8673-caa85810ae52" containerName="main" containerID="cri-o://2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096" gracePeriod=45 Jul 29 07:18:00.656330 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.656305 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:18:00.734278 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734246 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bnqf\" (UniqueName: \"kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734472 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734306 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734472 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734339 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734472 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734434 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734657 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734587 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734657 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734629 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir\") pod \"36e10614-c0e2-4269-8673-caa85810ae52\" (UID: \"36e10614-c0e2-4269-8673-caa85810ae52\") " Jul 29 07:18:00.734761 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734695 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home" (OuterVolumeSpecName: "home") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:00.734920 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734888 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache" (OuterVolumeSpecName: "model-cache") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:00.735011 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.734920 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:00.735150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.735132 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-home\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.735211 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.735157 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-model-cache\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.735211 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.735174 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-tmp-dir\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.736478 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.736458 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm" (OuterVolumeSpecName: "dshm") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:00.736995 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.736979 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf" (OuterVolumeSpecName: "kube-api-access-5bnqf") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "kube-api-access-5bnqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:00.736995 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.736986 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36e10614-c0e2-4269-8673-caa85810ae52" (UID: "36e10614-c0e2-4269-8673-caa85810ae52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:18:00.836281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.836254 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bnqf\" (UniqueName: \"kubernetes.io/projected/36e10614-c0e2-4269-8673-caa85810ae52-kube-api-access-5bnqf\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.836281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.836277 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e10614-c0e2-4269-8673-caa85810ae52-dshm\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.836281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.836287 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e10614-c0e2-4269-8673-caa85810ae52-tls-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:00.946329 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.946296 2556 generic.go:356] "Generic (PLEG): container finished" podID="36e10614-c0e2-4269-8673-caa85810ae52" containerID="2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096" exitCode=0 Jul 29 07:18:00.946538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.946331 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" event={"ID":"36e10614-c0e2-4269-8673-caa85810ae52","Type":"ContainerDied","Data":"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096"} Jul 29 07:18:00.946538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.946341 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" Jul 29 07:18:00.946538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.946370 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx" event={"ID":"36e10614-c0e2-4269-8673-caa85810ae52","Type":"ContainerDied","Data":"2d9c6e78799cdeeda1b55c33d2064defe1c895dd004236e3c4a89165229dc27b"} Jul 29 07:18:00.946538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.946409 2556 scope.go:117] "RemoveContainer" containerID="2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096" Jul 29 07:18:00.949045 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.949020 2556 generic.go:356] "Generic (PLEG): container finished" podID="44b0be51-f85a-4a17-910a-67dd831efa9d" containerID="32a0721de3bd5d3ce129ead923e92b31b1f5311b5bcfde1ee44ec4e9eddacfc2" exitCode=0 Jul 29 07:18:00.949182 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.949056 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762879488" event={"ID":"44b0be51-f85a-4a17-910a-67dd831efa9d","Type":"ContainerDied","Data":"32a0721de3bd5d3ce129ead923e92b31b1f5311b5bcfde1ee44ec4e9eddacfc2"} Jul 29 07:18:00.957499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.955717 2556 scope.go:117] "RemoveContainer" containerID="2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096" Jul 29 07:18:00.957499 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:18:00.956047 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096\": container with ID starting with 2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096 not found: ID does not exist" containerID="2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096" Jul 29 07:18:00.957499 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.956074 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096"} err="failed to get container status \"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096\": rpc error: code = NotFound desc = could not find container \"2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096\": container with ID starting with 2489fc3a7452f7a18a53a3f4e61a17e394caf7fe8ed19f28fe6518658113f096 not found: ID does not exist" Jul 29 07:18:00.979867 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.979834 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:18:00.983742 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:00.983723 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1e6a9b/rate-test-model-1e6a9b-kserve-7689558dc9-cwrsx"] Jul 29 07:18:01.929859 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:01.929826 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36e10614-c0e2-4269-8673-caa85810ae52" path="/var/lib/kubelet/pods/36e10614-c0e2-4269-8673-caa85810ae52/volumes" Jul 29 07:18:02.083059 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.083037 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.104088 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.104063 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659762879488_44b0be51-f85a-4a17-910a-67dd831efa9d/test-curl-7822-140659762879488/0.log" Jul 29 07:18:02.125392 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.125349 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:02.131530 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.131512 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:02.147117 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.147099 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ltst\" (UniqueName: \"kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst\") pod \"44b0be51-f85a-4a17-910a-67dd831efa9d\" (UID: \"44b0be51-f85a-4a17-910a-67dd831efa9d\") " Jul 29 07:18:02.149294 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.149265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst" (OuterVolumeSpecName: "kube-api-access-7ltst") pod "44b0be51-f85a-4a17-910a-67dd831efa9d" (UID: "44b0be51-f85a-4a17-910a-67dd831efa9d"). InnerVolumeSpecName "kube-api-access-7ltst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:02.248105 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.248067 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltst\" (UniqueName: \"kubernetes.io/projected/44b0be51-f85a-4a17-910a-67dd831efa9d-kube-api-access-7ltst\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:02.249679 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.249661 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:02.249984 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.249972 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0be51-f85a-4a17-910a-67dd831efa9d" containerName="test-curl-7822-140659762879488" Jul 29 07:18:02.250022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.249986 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0be51-f85a-4a17-910a-67dd831efa9d" containerName="test-curl-7822-140659762879488" Jul 29 07:18:02.250022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.249996 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36e10614-c0e2-4269-8673-caa85810ae52" containerName="main" Jul 29 07:18:02.250022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.250001 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="36e10614-c0e2-4269-8673-caa85810ae52" containerName="main" Jul 29 07:18:02.250116 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.250047 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="36e10614-c0e2-4269-8673-caa85810ae52" containerName="main" Jul 29 07:18:02.250116 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.250056 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b0be51-f85a-4a17-910a-67dd831efa9d" containerName="test-curl-7822-140659762879488" Jul 29 07:18:02.252749 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.252735 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.257848 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.257827 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:02.348966 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.348934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdwx\" (UniqueName: \"kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx\") pod \"test-curl-7822-140659762879488\" (UID: \"b11c7665-42f2-4ddd-be7b-f38a20201ac1\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.449969 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.449925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdwx\" (UniqueName: \"kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx\") pod \"test-curl-7822-140659762879488\" (UID: \"b11c7665-42f2-4ddd-be7b-f38a20201ac1\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.459028 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.458997 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdwx\" (UniqueName: \"kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx\") pod \"test-curl-7822-140659762879488\" (UID: \"b11c7665-42f2-4ddd-be7b-f38a20201ac1\") " pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.562486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.562398 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.692626 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.692600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:02.694571 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:18:02.694544 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11c7665_42f2_4ddd_be7b_f38a20201ac1.slice/crio-e48674b59a0d4ad8000a066924525a89fe845d55ff5c635fd508f54041050e2a WatchSource:0}: Error finding container e48674b59a0d4ad8000a066924525a89fe845d55ff5c635fd508f54041050e2a: Status 404 returned error can't find the container with id e48674b59a0d4ad8000a066924525a89fe845d55ff5c635fd508f54041050e2a Jul 29 07:18:02.960022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.959997 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de65c6e16936c0580d258580fa2643da9b4c466d2f0839263295456e58f4c1a" Jul 29 07:18:02.960022 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.960013 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:02.961104 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.961079 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762879488" event={"ID":"b11c7665-42f2-4ddd-be7b-f38a20201ac1","Type":"ContainerStarted","Data":"e48674b59a0d4ad8000a066924525a89fe845d55ff5c635fd508f54041050e2a"} Jul 29 07:18:02.971611 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:02.971589 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7822-140659762879488" oldPodUID="44b0be51-f85a-4a17-910a-67dd831efa9d" podUID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" Jul 29 07:18:03.928680 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:03.928646 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b0be51-f85a-4a17-910a-67dd831efa9d" path="/var/lib/kubelet/pods/44b0be51-f85a-4a17-910a-67dd831efa9d/volumes" Jul 29 07:18:03.967523 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:03.967499 2556 generic.go:356] "Generic (PLEG): container finished" podID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" containerID="3c19362ea27469afd1c4c14cdfd7c52f6a1cc11d0d5e9bd2bf690405d6e1b73c" exitCode=0 Jul 29 07:18:03.967887 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:03.967582 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659762879488" event={"ID":"b11c7665-42f2-4ddd-be7b-f38a20201ac1","Type":"ContainerDied","Data":"3c19362ea27469afd1c4c14cdfd7c52f6a1cc11d0d5e9bd2bf690405d6e1b73c"} Jul 29 07:18:05.106396 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.106359 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:05.125202 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.125178 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659762879488_b11c7665-42f2-4ddd-be7b-f38a20201ac1/test-curl-7822-140659762879488/0.log" Jul 29 07:18:05.146941 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.146908 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:05.149345 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.149318 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659762879488"] Jul 29 07:18:05.176928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.176901 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmdwx\" (UniqueName: \"kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx\") pod \"b11c7665-42f2-4ddd-be7b-f38a20201ac1\" (UID: \"b11c7665-42f2-4ddd-be7b-f38a20201ac1\") " Jul 29 07:18:05.179190 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.179168 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx" (OuterVolumeSpecName: "kube-api-access-gmdwx") pod "b11c7665-42f2-4ddd-be7b-f38a20201ac1" (UID: "b11c7665-42f2-4ddd-be7b-f38a20201ac1"). InnerVolumeSpecName "kube-api-access-gmdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:05.277775 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.277702 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdwx\" (UniqueName: \"kubernetes.io/projected/b11c7665-42f2-4ddd-be7b-f38a20201ac1-kube-api-access-gmdwx\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:18:05.644301 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.644223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:05.928184 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.928110 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" path="/var/lib/kubelet/pods/b11c7665-42f2-4ddd-be7b-f38a20201ac1/volumes" Jul 29 07:18:05.935652 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.935626 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:18:05.936069 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.936050 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" containerName="test-curl-7822-140659762879488" Jul 29 07:18:05.936069 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.936069 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" containerName="test-curl-7822-140659762879488" Jul 29 07:18:05.936194 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.936147 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11c7665-42f2-4ddd-be7b-f38a20201ac1" containerName="test-curl-7822-140659762879488" Jul 29 07:18:05.939194 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.939178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.941896 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.941881 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-aab086-openshift-default-dockercfg-grxqp\"" Jul 29 07:18:05.949927 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.949902 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:18:05.951028 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.951001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:05.978731 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.978713 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659762879488" Jul 29 07:18:05.978828 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.978773 2556 scope.go:117] "RemoveContainer" containerID="3c19362ea27469afd1c4c14cdfd7c52f6a1cc11d0d5e9bd2bf690405d6e1b73c" Jul 29 07:18:05.983368 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983349 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb96\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983475 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983475 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983454 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983475 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983471 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983609 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983539 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983609 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983589 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983688 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983627 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983688 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983672 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:05.983762 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:05.983728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085096 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085061 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085096 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb96\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085312 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085420 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085734 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085570 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085734 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085828 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.085992 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085964 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.086125 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.085973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.087849 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.087814 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.088040 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.088024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.093635 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.093614 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.093767 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.093749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb96\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96\") pod \"e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.249451 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.249412 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:06.279479 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.279437 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:06.387876 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.387847 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:18:06.389162 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:18:06.389131 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcb9fe5_0959_4061_b28e_3b87f60ad13c.slice/crio-c49900bf4d81be706a2cbe9b36bd8e7e8b778e9d8ca381aea1d2a26e3cbdcea3 WatchSource:0}: Error finding container c49900bf4d81be706a2cbe9b36bd8e7e8b778e9d8ca381aea1d2a26e3cbdcea3: Status 404 returned error can't find the container with id c49900bf4d81be706a2cbe9b36bd8e7e8b778e9d8ca381aea1d2a26e3cbdcea3 Jul 29 07:18:06.391367 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.391337 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:06.391476 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.391416 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:06.391476 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.391445 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:06.986314 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.986273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerStarted","Data":"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744"} Jul 29 07:18:06.986512 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:06.986340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerStarted","Data":"c49900bf4d81be706a2cbe9b36bd8e7e8b778e9d8ca381aea1d2a26e3cbdcea3"} Jul 29 07:18:07.009144 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:07.009098 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podStartSLOduration=2.009083184 podStartE2EDuration="2.009083184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:07.008261721 +0000 UTC m=+3331.542341294" watchObservedRunningTime="2026-07-29 07:18:07.009083184 +0000 UTC m=+3331.543162757" Jul 29 07:18:07.250326 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:07.250243 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:07.255167 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:07.255143 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:07.990437 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:07.990407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:07.991507 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:07.991490 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:08.944666 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:08.944626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:09.044344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:09.044299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:09.205699 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:09.205616 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:09.497611 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:09.497521 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:09.718495 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:09.718462 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:09.821828 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:09.821730 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:12.393486 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:12.393449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:27.082349 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.082300 2556 generic.go:356] "Generic (PLEG): container finished" podID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerID="61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744" exitCode=0 Jul 29 07:18:27.082797 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.082367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerDied","Data":"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744"} Jul 29 07:18:27.082797 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.082757 2556 scope.go:117] "RemoveContainer" containerID="61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744" Jul 29 07:18:27.083972 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.083943 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:27.084039 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.084025 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:27.084079 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:27.084060 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:28.088511 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:28.088477 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerStarted","Data":"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272"} Jul 29 07:18:28.250803 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:28.250764 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:28.251004 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:28.250887 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 29 07:18:28.251004 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:28.250940 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 29 07:18:29.249789 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:29.249744 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 29 07:18:29.250285 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:29.249825 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 29 07:18:30.250621 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:30.250574 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 29 07:18:30.251129 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:30.250657 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 29 07:18:31.250870 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:31.250831 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 29 07:18:31.251507 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:31.251475 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 29 07:18:32.253406 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:32.253362 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:32.253796 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:32.253630 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:32.254263 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:32.254243 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:18:33.942073 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:33.942018 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:34.671038 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:34.670999 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:34.967209 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:34.967182 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:18:34.969617 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:34.969598 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:34.975042 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:34.975018 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-04c3b9-openshift-default-dockercfg-669j8\"" Jul 29 07:18:34.989510 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:34.989488 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:18:35.023187 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.023156 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:35.157180 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157308 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157346 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157427 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7sh\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.157655 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.157546 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.223120 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.223028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:35.258771 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.258895 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258802 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.258895 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258843 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.258895 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.258895 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259099 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7sh\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259099 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258945 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259099 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.258982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259099 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259293 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259351 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259319 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259500 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259474 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259608 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.259741 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.259718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.261331 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.261316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.261722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.261705 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.267562 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.267527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.267663 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.267631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7sh\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh\") pod \"e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.279973 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.279949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:35.417614 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.417587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:18:35.419261 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:18:35.419226 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5722fdb_cdbc_4cba_ae48_8a8cbeeff337.slice/crio-6ecde28bac5844ea8b1b49670fc029fdec1f8f2a3466b5400e19bce797ef419c WatchSource:0}: Error finding container 6ecde28bac5844ea8b1b49670fc029fdec1f8f2a3466b5400e19bce797ef419c: Status 404 returned error can't find the container with id 6ecde28bac5844ea8b1b49670fc029fdec1f8f2a3466b5400e19bce797ef419c Jul 29 07:18:35.421590 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.421562 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:35.421654 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.421625 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:35.421654 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:35.421651 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:36.131320 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:36.131284 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerStarted","Data":"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2"} Jul 29 07:18:36.131723 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:36.131337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerStarted","Data":"6ecde28bac5844ea8b1b49670fc029fdec1f8f2a3466b5400e19bce797ef419c"} Jul 29 07:18:36.154584 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:36.154531 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podStartSLOduration=2.154511779 podStartE2EDuration="2.154511779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:36.152330444 +0000 UTC m=+3360.686410086" watchObservedRunningTime="2026-07-29 07:18:36.154511779 +0000 UTC m=+3360.688591351" Jul 29 07:18:36.280268 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:36.280225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:36.285172 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:36.285148 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:37.136089 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:37.136050 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:37.137059 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:37.137040 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:38.070722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.070670 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:38.175355 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.175312 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:38.272527 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.272495 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:38.412128 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.412039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:38.889246 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.889206 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:38.986933 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:38.986893 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:56.226622 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.226588 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerID="cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2" exitCode=0 Jul 29 07:18:56.227156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.226669 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerDied","Data":"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2"} Jul 29 07:18:56.227227 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.227192 2556 scope.go:117] "RemoveContainer" containerID="cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2" Jul 29 07:18:56.228323 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.228298 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:56.228411 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.228364 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:56.228461 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:56.228410 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 07:18:57.233078 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:57.233044 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerStarted","Data":"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f"} Jul 29 07:18:57.280749 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:57.280712 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:18:57.280975 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:57.280949 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 07:18:57.281035 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:57.281017 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 07:18:58.280864 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:58.280831 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 07:18:58.281242 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:58.280893 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 07:18:59.280288 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:59.280256 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 07:18:59.280493 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:59.280310 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 07:18:59.368484 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:59.368430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:18:59.478912 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:18:59.478864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:00.280559 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:00.280522 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 07:19:00.280735 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:00.280580 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 07:19:01.287636 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:01.287598 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:19:01.288048 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:01.287885 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:19:01.288722 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:01.288707 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:19:03.030515 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:03.030481 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:19:16.963252 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:19:16.963219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:00.634202 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:00.634154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:02.345679 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:02.345641 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:07.313976 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:07.313940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:07.415055 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:07.415002 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:07.582520 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:07.582419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:07.901807 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:07.901706 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:08.065432 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:08.065394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:08.163617 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:08.163527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:32.401150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:32.401101 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:48.138912 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:48.138870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:48.226876 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:48.226832 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:48.324928 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:48.324890 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:48.430411 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:48.430296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:48.753664 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:48.753629 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:54.716286 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:54.716255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:54.819956 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:54.819925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:55.029019 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:55.028930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:20:55.265601 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:20:55.265555 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:14.555354 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:14.555270 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:14.639277 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:14.639246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:14.715435 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:14.715401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:14.805520 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:14.805430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:15.153541 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:15.153417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:20.230323 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:20.230280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:56.774257 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:56.774223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:21:57.231014 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:21:57.230980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:02.888323 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:02.888276 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:02.959887 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:02.959854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:03.172720 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:03.172632 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:22:03.172891 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:03.172857 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" containerID="cri-o://50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272" gracePeriod=30 Jul 29 07:22:03.238134 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:03.238097 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:03.456994 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:03.456957 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:08.436070 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.436007 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:22:08.509433 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509403 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509606 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509442 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfb96\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509606 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509470 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509606 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509517 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509606 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509587 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509786 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509630 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509786 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509665 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.509786 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.510141 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510076 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket\") pod \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\" (UID: \"cbcb9fe5-0959-4061-b28e-3b87f60ad13c\") " Jul 29 07:22:08.510141 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509926 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:08.510141 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.509968 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:08.510410 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510093 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:08.510481 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510438 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:08.510652 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510632 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.510729 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510656 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.510729 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510669 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.510729 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510698 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.510929 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.510907 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:08.511887 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.511863 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:08.511887 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.511876 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:08.512173 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.512147 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:22:08.512266 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.512154 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96" (OuterVolumeSpecName: "kube-api-access-cfb96") pod "cbcb9fe5-0959-4061-b28e-3b87f60ad13c" (UID: "cbcb9fe5-0959-4061-b28e-3b87f60ad13c"). InnerVolumeSpecName "kube-api-access-cfb96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:08.611954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.611904 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.611954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.611951 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.611954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.611961 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb96\" (UniqueName: \"kubernetes.io/projected/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-kube-api-access-cfb96\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.611954 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.611971 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.612223 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.611979 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcb9fe5-0959-4061-b28e-3b87f60ad13c-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:08.892562 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.892522 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:08.993731 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:08.993702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:09.093053 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.093023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:09.137886 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.137848 2556 generic.go:356] "Generic (PLEG): container finished" podID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerID="50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272" exitCode=0 Jul 29 07:22:09.138095 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.137893 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerDied","Data":"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272"} Jul 29 07:22:09.138095 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.137962 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" event={"ID":"cbcb9fe5-0959-4061-b28e-3b87f60ad13c","Type":"ContainerDied","Data":"c49900bf4d81be706a2cbe9b36bd8e7e8b778e9d8ca381aea1d2a26e3cbdcea3"} Jul 29 07:22:09.138095 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.137977 2556 scope.go:117] "RemoveContainer" containerID="50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272" Jul 29 07:22:09.138095 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.137906 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz" Jul 29 07:22:09.147818 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.147798 2556 scope.go:117] "RemoveContainer" containerID="61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744" Jul 29 07:22:09.156819 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.156798 2556 scope.go:117] "RemoveContainer" containerID="50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272" Jul 29 07:22:09.157092 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:09.157073 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272\": container with ID starting with 50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272 not found: ID does not exist" containerID="50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272" Jul 29 07:22:09.157142 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.157101 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272"} err="failed to get container status \"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272\": rpc error: code = NotFound desc = could not find container \"50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272\": container with ID starting with 50ca24e1e0a1c6fcd99d75e6a99ed61ba2cf02277caa804b49a32301ed46e272 not found: ID does not exist" Jul 29 07:22:09.157142 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.157118 2556 scope.go:117] "RemoveContainer" containerID="61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744" Jul 29 07:22:09.157355 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:09.157341 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744\": container with ID starting with 61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744 not found: ID does not exist" containerID="61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744" Jul 29 07:22:09.157494 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.157359 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744"} err="failed to get container status \"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744\": rpc error: code = NotFound desc = could not find container \"61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744\": container with ID starting with 61d428b2e9552704d98f59f5c21cfba4fcb4530bf97ba3636d4f98bf4a0c2744 not found: ID does not exist" Jul 29 07:22:09.162423 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.162395 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:22:09.166924 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.166902 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-aab086-openshift-default-6d8cc8986-8s9nz"] Jul 29 07:22:09.170566 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.170543 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:09.488832 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.488798 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:09.927820 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:09.927741 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" path="/var/lib/kubelet/pods/cbcb9fe5-0959-4061-b28e-3b87f60ad13c/volumes" Jul 29 07:22:13.917620 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:13.917576 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:15.581774 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:15.581739 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:15.655670 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:15.655635 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:15.857119 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:15.857027 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:22:15.857315 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:15.857289 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" containerID="cri-o://d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f" gracePeriod=30 Jul 29 07:22:15.875239 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:15.875207 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:16.173023 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:16.172932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:16.290199 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:16.290157 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:22:16.290359 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:16.290221 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:22:20.860204 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:20.860162 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:21.047984 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.047855 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:21.117608 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.117538 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:22:21.195192 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.195151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:21.196018 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.195977 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerID="d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f" exitCode=0 Jul 29 07:22:21.196156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.196034 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerDied","Data":"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f"} Jul 29 07:22:21.196156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.196068 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" Jul 29 07:22:21.196156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.196098 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4" event={"ID":"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337","Type":"ContainerDied","Data":"6ecde28bac5844ea8b1b49670fc029fdec1f8f2a3466b5400e19bce797ef419c"} Jul 29 07:22:21.196156 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.196130 2556 scope.go:117] "RemoveContainer" containerID="d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f" Jul 29 07:22:21.204789 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.204767 2556 scope.go:117] "RemoveContainer" containerID="cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2" Jul 29 07:22:21.214613 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.214487 2556 scope.go:117] "RemoveContainer" containerID="d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f" Jul 29 07:22:21.215201 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:21.215059 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f\": container with ID starting with d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f not found: ID does not exist" containerID="d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f" Jul 29 07:22:21.215201 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.215106 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f"} err="failed to get container status \"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f\": rpc error: code = NotFound desc = could not find container \"d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f\": container with ID starting with d81d5ae350e4c0a051b847c9eaeb0adc451b989cb807a39182970aaf3d131d3f not found: ID does not exist" Jul 29 07:22:21.215201 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.215130 2556 scope.go:117] "RemoveContainer" containerID="cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2" Jul 29 07:22:21.215576 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:21.215508 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2\": container with ID starting with cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2 not found: ID does not exist" containerID="cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2" Jul 29 07:22:21.215576 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.215543 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2"} err="failed to get container status \"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2\": rpc error: code = NotFound desc = could not find container \"cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2\": container with ID starting with cf5261f5406c1f0f18fa50c691073366a88d22c8a168da6b1db9619036767ad2 not found: ID does not exist" Jul 29 07:22:21.240249 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240219 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240314 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240340 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240400 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240425 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240424 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240647 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240446 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240647 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240478 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240647 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240515 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240647 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240550 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7sh\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh\") pod \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\" (UID: \"e5722fdb-cdbc-4cba-ae48-8a8cbeeff337\") " Jul 29 07:22:21.240841 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240697 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:21.240841 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240785 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:21.240961 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240929 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.240961 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240946 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.241056 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.240983 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:21.241117 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.241056 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:21.241281 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.241259 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:21.242953 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.242924 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:22:21.243059 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.243042 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh" (OuterVolumeSpecName: "kube-api-access-6r7sh") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "kube-api-access-6r7sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:21.243230 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.243209 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:21.243334 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.243220 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" (UID: "e5722fdb-cdbc-4cba-ae48-8a8cbeeff337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:21.262066 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.262035 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:21.341659 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341620 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341659 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341652 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341659 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341661 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341659 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341669 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341932 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341677 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341932 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341684 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7sh\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-kube-api-access-6r7sh\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.341932 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.341695 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:21.519215 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.519182 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:22:21.523146 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.523112 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-04c3b9-openshift-default-54595bbbc4-t7ct4"] Jul 29 07:22:21.537817 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.537786 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:21.928048 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:21.927964 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" path="/var/lib/kubelet/pods/e5722fdb-cdbc-4cba-ae48-8a8cbeeff337/volumes" Jul 29 07:22:27.402618 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.402535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:27.500944 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.500907 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:27.702032 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.701990 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:22:27.702270 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.702243 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" containerID="cri-o://84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe" gracePeriod=30 Jul 29 07:22:27.754575 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.754537 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:27.934141 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:27.934106 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:32.949258 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:32.949234 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:22:33.041522 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041443 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041522 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041483 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041522 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041509 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041528 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041551 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041575 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041625 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnpml\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041650 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.041790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041678 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs\") pod \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\" (UID: \"6588ad46-c63f-4453-aea4-9ad36cfcf04e\") " Jul 29 07:22:33.042090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041825 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:33.042090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041915 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:33.042090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041982 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:33.042090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.041989 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.042090 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.042067 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.042297 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.042080 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:33.042974 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.042941 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:33.043908 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.043887 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:33.043995 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.043921 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:22:33.044150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.044127 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:33.044150 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.044141 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml" (OuterVolumeSpecName: "kube-api-access-tnpml") pod "6588ad46-c63f-4453-aea4-9ad36cfcf04e" (UID: "6588ad46-c63f-4453-aea4-9ad36cfcf04e"). InnerVolumeSpecName "kube-api-access-tnpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:33.142842 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142806 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.142842 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142836 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.142842 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142846 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.143036 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142853 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.143036 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142861 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpml\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-kube-api-access-tnpml\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.143036 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142869 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6588ad46-c63f-4453-aea4-9ad36cfcf04e-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.143036 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.142877 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6588ad46-c63f-4453-aea4-9ad36cfcf04e-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:33.254353 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.254316 2556 generic.go:356] "Generic (PLEG): container finished" podID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerID="84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe" exitCode=0 Jul 29 07:22:33.254538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.254364 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" Jul 29 07:22:33.254538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.254399 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerDied","Data":"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe"} Jul 29 07:22:33.254538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.254453 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5" event={"ID":"6588ad46-c63f-4453-aea4-9ad36cfcf04e","Type":"ContainerDied","Data":"36ef793cf3aef5013a2a3665a820593e16a5527dc7ac14a4c0c8e8b0e9afbc10"} Jul 29 07:22:33.254538 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.254475 2556 scope.go:117] "RemoveContainer" containerID="84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe" Jul 29 07:22:33.263516 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.263499 2556 scope.go:117] "RemoveContainer" containerID="dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52" Jul 29 07:22:33.270969 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.270955 2556 scope.go:117] "RemoveContainer" containerID="84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe" Jul 29 07:22:33.271204 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:33.271187 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe\": container with ID starting with 84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe not found: ID does not exist" containerID="84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe" Jul 29 07:22:33.271269 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.271210 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe"} err="failed to get container status \"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe\": rpc error: code = NotFound desc = could not find container \"84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe\": container with ID starting with 84bb954784cfd8156c48eb60e1e0c63b816fdb101553deed2d32121d882f2ebe not found: ID does not exist" Jul 29 07:22:33.271269 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.271225 2556 scope.go:117] "RemoveContainer" containerID="dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52" Jul 29 07:22:33.271482 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:33.271467 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52\": container with ID starting with dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52 not found: ID does not exist" containerID="dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52" Jul 29 07:22:33.271527 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.271487 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52"} err="failed to get container status \"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52\": rpc error: code = NotFound desc = could not find container \"dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52\": container with ID starting with dc511ce07072ff58596e429be8b70f6bbabc9d55e6f0e503edca7dcfb949bf52 not found: ID does not exist" Jul 29 07:22:33.276610 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.276590 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:22:33.280275 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.280254 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-50e693-openshift-default-75fdf78ff6-fhlc5"] Jul 29 07:22:33.927965 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:33.927929 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" path="/var/lib/kubelet/pods/6588ad46-c63f-4453-aea4-9ad36cfcf04e/volumes" Jul 29 07:22:36.874329 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:36.874285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:38.660563 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:38.660528 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:38.741947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:38.741911 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:38.816958 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:38.816918 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:38.886305 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:38.886269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:39.219139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:39.219098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:45.541670 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:45.541638 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:45.642451 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:45.642412 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:45.778047 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:45.778011 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:22:45.778309 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:45.778284 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" containerID="cri-o://f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817" gracePeriod=30 Jul 29 07:22:45.806236 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:45.806139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:46.089276 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:46.089190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:22:51.034951 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.034926 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:22:51.093464 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093361 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093464 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093448 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093668 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093484 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093668 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093520 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093668 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093574 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093668 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093621 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093668 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093647 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cczh\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093911 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093675 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093911 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093709 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token\") pod \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\" (UID: \"85b68fe7-3036-435c-89ab-58ad7e3c25c8\") " Jul 29 07:22:51.093911 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.093829 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:51.094268 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.094239 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.094410 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.094310 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:51.094483 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.094400 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:51.094483 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.094444 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:51.095264 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.095176 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:51.096440 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.096397 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:22:51.096894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.096861 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:51.096894 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.096877 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh" (OuterVolumeSpecName: "kube-api-access-9cczh") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "kube-api-access-9cczh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:51.097068 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.096912 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "85b68fe7-3036-435c-89ab-58ad7e3c25c8" (UID: "85b68fe7-3036-435c-89ab-58ad7e3c25c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:51.194947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194918 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-credential-socket\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.194947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194942 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-podinfo\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.194947 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194952 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-envoy\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.195159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194959 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istiod-ca-cert\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.195159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194968 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-workload-certs\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.195159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194976 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cczh\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-kube-api-access-9cczh\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.195159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194984 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-data\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.195159 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.194993 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b68fe7-3036-435c-89ab-58ad7e3c25c8-istio-token\") on node \"ip-10-0-131-31.ec2.internal\" DevicePath \"\"" Jul 29 07:22:51.341370 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.341336 2556 generic.go:356] "Generic (PLEG): container finished" podID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerID="f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817" exitCode=0 Jul 29 07:22:51.341558 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.341410 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" Jul 29 07:22:51.341558 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.341428 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerDied","Data":"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817"} Jul 29 07:22:51.341558 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.341492 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld" event={"ID":"85b68fe7-3036-435c-89ab-58ad7e3c25c8","Type":"ContainerDied","Data":"c85fe54f30b1f3e1a8d861eda70d7c1e30426c20669032123e48d4b7d1d96eb7"} Jul 29 07:22:51.341558 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.341528 2556 scope.go:117] "RemoveContainer" containerID="f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817" Jul 29 07:22:51.350469 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.350448 2556 scope.go:117] "RemoveContainer" containerID="ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594" Jul 29 07:22:51.358571 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.358553 2556 scope.go:117] "RemoveContainer" containerID="f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817" Jul 29 07:22:51.358802 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:51.358784 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817\": container with ID starting with f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817 not found: ID does not exist" containerID="f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817" Jul 29 07:22:51.358846 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.358810 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817"} err="failed to get container status \"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817\": rpc error: code = NotFound desc = could not find container \"f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817\": container with ID starting with f5c2f099887ef82339b427cb395119188141601cb39a02434863a3ea3fee2817 not found: ID does not exist" Jul 29 07:22:51.358846 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.358830 2556 scope.go:117] "RemoveContainer" containerID="ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594" Jul 29 07:22:51.359069 ip-10-0-131-31 kubenswrapper[2556]: E0729 07:22:51.359052 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594\": container with ID starting with ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594 not found: ID does not exist" containerID="ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594" Jul 29 07:22:51.359127 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.359075 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594"} err="failed to get container status \"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594\": rpc error: code = NotFound desc = could not find container \"ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594\": container with ID starting with ad182c5a49b05c0791d9f01ef05a8fb9e12c280e27c615302b5b1e6ca7b8c594 not found: ID does not exist" Jul 29 07:22:51.371429 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.371407 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:22:51.374875 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.374855 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1e6a9b-openshift-default-5bcfdcdfbd-flpld"] Jul 29 07:22:51.928296 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:51.928265 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" path="/var/lib/kubelet/pods/85b68fe7-3036-435c-89ab-58ad7e3c25c8/volumes" Jul 29 07:22:52.991205 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:52.991175 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-fvrsb_05fcf4a5-b597-448f-b6a8-7163b05fdc75/copy-manifests/0.log" Jul 29 07:22:53.004052 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:53.004022 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-fvrsb_05fcf4a5-b597-448f-b6a8-7163b05fdc75/manager/0.log" Jul 29 07:22:55.442496 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:55.442464 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-895pz_cc14b502-a312-4b50-a82d-3a9c3c158d91/manager/0.log" Jul 29 07:22:55.662457 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:55.662428 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8w2tr_2d289693-1ca5-4e45-ba67-6ae353f4cf1f/kuadrant-console-plugin/0.log" Jul 29 07:22:55.794342 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:55.794258 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lnsgw_e8722a78-6392-4173-a236-eaba28aeaa4d/registry-server/0.log" Jul 29 07:22:56.144591 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:56.144513 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-s2lnr_0a85deae-4bc9-4d36-a487-bc132837791c/manager/0.log" Jul 29 07:22:56.498246 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:56.498217 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cn74bt_88979fb6-2d23-4cf3-ae48-c41f48e23238/istio-proxy/0.log" Jul 29 07:22:56.960456 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:56.960416 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/1.log" Jul 29 07:22:57.763058 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:57.763026 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-t7jjx_98ddf9f5-19b7-4d93-b156-27db15d3db0b/main/0.log" Jul 29 07:22:58.110905 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:58.110823 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ghfst_ef65fa25-7948-490a-9003-ab78da495b65/main/0.log" Jul 29 07:22:58.235587 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:22:58.235559 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jf9fg_6b89e206-30a3-4462-b629-262e6dfba4e9/main/0.log" Jul 29 07:23:01.589762 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:01.589718 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:01.682000 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:01.681970 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:03.770513 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770475 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rtf28/must-gather-57bpx"] Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770898 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770915 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770931 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770939 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770953 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.770967 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770961 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770972 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770983 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.770996 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771004 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771016 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771025 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771036 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771045 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771125 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771138 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771149 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771159 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771169 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771178 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b68fe7-3036-435c-89ab-58ad7e3c25c8" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771187 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6588ad46-c63f-4453-aea4-9ad36cfcf04e" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771273 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.771291 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771282 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5722fdb-cdbc-4cba-ae48-8a8cbeeff337" containerName="istio-proxy" Jul 29 07:23:03.772124 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.771353 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcb9fe5-0959-4061-b28e-3b87f60ad13c" containerName="istio-proxy" Jul 29 07:23:03.775999 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.775979 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:03.778466 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.778432 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:03.779208 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.779186 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rtf28\"/\"default-dockercfg-zbqqp\"" Jul 29 07:23:03.780633 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.780619 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rtf28\"/\"kube-root-ca.crt\"" Jul 29 07:23:03.781623 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.781600 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rtf28\"/\"openshift-service-ca.crt\"" Jul 29 07:23:03.786824 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.786803 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/must-gather-57bpx"] Jul 29 07:23:03.896300 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.896266 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" limit=1024 Jul 29 07:23:03.902409 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.902353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmfh\" (UniqueName: \"kubernetes.io/projected/ce16bd20-e5a4-49dd-a916-413e87c66dbf-kube-api-access-jrmfh\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:03.902553 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:03.902536 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce16bd20-e5a4-49dd-a916-413e87c66dbf-must-gather-output\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.003547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.003503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce16bd20-e5a4-49dd-a916-413e87c66dbf-must-gather-output\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.003547 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.003553 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmfh\" (UniqueName: \"kubernetes.io/projected/ce16bd20-e5a4-49dd-a916-413e87c66dbf-kube-api-access-jrmfh\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.003862 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.003812 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce16bd20-e5a4-49dd-a916-413e87c66dbf-must-gather-output\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.018191 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.018166 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmfh\" (UniqueName: \"kubernetes.io/projected/ce16bd20-e5a4-49dd-a916-413e87c66dbf-kube-api-access-jrmfh\") pod \"must-gather-57bpx\" (UID: \"ce16bd20-e5a4-49dd-a916-413e87c66dbf\") " pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.085165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.085086 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/must-gather-57bpx" Jul 29 07:23:04.254193 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:23:04.254162 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce16bd20_e5a4_49dd_a916_413e87c66dbf.slice/crio-0ba6f478bc3102a99de2ed70400e3651c982fe19a0e7a4327c9278f6810b867f WatchSource:0}: Error finding container 0ba6f478bc3102a99de2ed70400e3651c982fe19a0e7a4327c9278f6810b867f: Status 404 returned error can't find the container with id 0ba6f478bc3102a99de2ed70400e3651c982fe19a0e7a4327c9278f6810b867f Jul 29 07:23:04.254704 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.254682 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/must-gather-57bpx"] Jul 29 07:23:04.256024 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.256004 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:23:04.400057 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:04.399977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/must-gather-57bpx" event={"ID":"ce16bd20-e5a4-49dd-a916-413e87c66dbf","Type":"ContainerStarted","Data":"0ba6f478bc3102a99de2ed70400e3651c982fe19a0e7a4327c9278f6810b867f"} Jul 29 07:23:05.408825 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:05.408730 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/must-gather-57bpx" event={"ID":"ce16bd20-e5a4-49dd-a916-413e87c66dbf","Type":"ContainerStarted","Data":"242a0d552f988779fa0b6cead3f2b1294f9c75e35744f3f04a106cc3afd4508c"} Jul 29 07:23:05.408825 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:05.408787 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/must-gather-57bpx" event={"ID":"ce16bd20-e5a4-49dd-a916-413e87c66dbf","Type":"ContainerStarted","Data":"d1915fcf081b5b598bf58c4ee69f8e808686b13af89735887d9f58c99da01eaa"} Jul 29 07:23:05.433165 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:05.433116 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rtf28/must-gather-57bpx" podStartSLOduration=1.744643851 podStartE2EDuration="2.433101904s" totalImagesPullingTime="688.458053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:05.432834694 +0000 UTC m=+3629.966914270" watchObservedRunningTime="2026-07-29 07:23:05.433101904 +0000 UTC m=+3629.967181477" Jul 29 07:23:06.800367 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:06.800322 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dg9h9_b21ba907-5748-4c3f-b4d6-d8ee66073526/global-pull-secret-syncer/0.log" Jul 29 07:23:07.055456 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:07.055227 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c2ghx_5fd37587-97c4-4ee5-9a0c-db3d92b3f4a1/konnectivity-agent/0.log" Jul 29 07:23:07.291507 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:07.291478 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-31.ec2.internal_664c079835b1aca0098a8fa9d9f98d07/haproxy/0.log" Jul 29 07:23:12.846344 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:12.846257 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-895pz_cc14b502-a312-4b50-a82d-3a9c3c158d91/manager/0.log" Jul 29 07:23:12.940761 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:12.940728 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8w2tr_2d289693-1ca5-4e45-ba67-6ae353f4cf1f/kuadrant-console-plugin/0.log" Jul 29 07:23:13.006978 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:13.006940 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lnsgw_e8722a78-6392-4173-a236-eaba28aeaa4d/registry-server/0.log" Jul 29 07:23:13.521470 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:13.521422 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-s2lnr_0a85deae-4bc9-4d36-a487-bc132837791c/manager/0.log" Jul 29 07:23:15.689333 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:15.689294 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fcc59db9d-x9mc5_30860552-c0c5-4a08-b690-1bcab249879d/metrics-server/0.log" Jul 29 07:23:15.796506 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:15.796475 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dxgz_a896ceac-3ce1-45c5-ac59-850c78696ebe/node-exporter/0.log" Jul 29 07:23:15.857952 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:15.857915 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dxgz_a896ceac-3ce1-45c5-ac59-850c78696ebe/kube-rbac-proxy/0.log" Jul 29 07:23:15.902697 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:15.902668 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8dxgz_a896ceac-3ce1-45c5-ac59-850c78696ebe/init-textfile/0.log" Jul 29 07:23:16.221329 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.221285 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr"] Jul 29 07:23:16.225787 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.225761 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.248600 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.248569 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr"] Jul 29 07:23:16.331609 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.331564 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-sys\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.331802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.331614 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-proc\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.331802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.331672 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-podres\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.331802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.331749 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbq8n\" (UniqueName: \"kubernetes.io/projected/526593ae-75e3-4692-92b7-a7154cb50bda-kube-api-access-wbq8n\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.331802 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.331795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-lib-modules\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.432782 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.432741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-podres\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.432939 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.432862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbq8n\" (UniqueName: \"kubernetes.io/projected/526593ae-75e3-4692-92b7-a7154cb50bda-kube-api-access-wbq8n\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.432939 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.432906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-podres\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.432939 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.432907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-lib-modules\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.433092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.432979 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-sys\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.433092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.433017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-lib-modules\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.433092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.433024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-proc\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.433092 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.433070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-proc\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.433278 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.433119 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526593ae-75e3-4692-92b7-a7154cb50bda-sys\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.443390 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.443345 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbq8n\" (UniqueName: \"kubernetes.io/projected/526593ae-75e3-4692-92b7-a7154cb50bda-kube-api-access-wbq8n\") pod \"perf-node-gather-daemonset-kq7nr\" (UID: \"526593ae-75e3-4692-92b7-a7154cb50bda\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.539313 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.539217 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:16.696164 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.696058 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr"] Jul 29 07:23:16.699035 ip-10-0-131-31 kubenswrapper[2556]: W0729 07:23:16.699007 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod526593ae_75e3_4692_92b7_a7154cb50bda.slice/crio-a2fdfe097b5de17a02666d0d7fa7f38aa3a71b845523c8c750222cebb6fa033d WatchSource:0}: Error finding container a2fdfe097b5de17a02666d0d7fa7f38aa3a71b845523c8c750222cebb6fa033d: Status 404 returned error can't find the container with id a2fdfe097b5de17a02666d0d7fa7f38aa3a71b845523c8c750222cebb6fa033d Jul 29 07:23:16.766221 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.766117 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f96b55844-g6snr_9f8b113d-073c-4eb0-aedc-1f77f1e496eb/telemeter-client/0.log" Jul 29 07:23:16.798790 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.798699 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f96b55844-g6snr_9f8b113d-073c-4eb0-aedc-1f77f1e496eb/reload/0.log" Jul 29 07:23:16.840446 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:16.840363 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f96b55844-g6snr_9f8b113d-073c-4eb0-aedc-1f77f1e496eb/kube-rbac-proxy/0.log" Jul 29 07:23:17.486945 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:17.486897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" event={"ID":"526593ae-75e3-4692-92b7-a7154cb50bda","Type":"ContainerStarted","Data":"b520ec20ebd065cbd2dc78973341f8b24284347b548ddf7add1db547b46ab939"} Jul 29 07:23:17.487139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:17.486963 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" event={"ID":"526593ae-75e3-4692-92b7-a7154cb50bda","Type":"ContainerStarted","Data":"a2fdfe097b5de17a02666d0d7fa7f38aa3a71b845523c8c750222cebb6fa033d"} Jul 29 07:23:17.487139 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:17.486995 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:17.517585 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:17.517511 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" podStartSLOduration=1.517488255 podStartE2EDuration="1.517488255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:17.513970678 +0000 UTC m=+3642.048050252" watchObservedRunningTime="2026-07-29 07:23:17.517488255 +0000 UTC m=+3642.051567840" Jul 29 07:23:21.253917 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:21.253890 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9ltd_ce8b77c4-2fc8-4bd3-891d-94f96aca97d9/dns/0.log" Jul 29 07:23:21.307181 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:21.307153 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9ltd_ce8b77c4-2fc8-4bd3-891d-94f96aca97d9/kube-rbac-proxy/0.log" Jul 29 07:23:21.405021 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:21.404993 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg47d_cfad06fb-db01-4906-baf6-92c769990810/dns-node-resolver/0.log" Jul 29 07:23:22.033341 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:22.033308 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-566654d898-kc6wh_12733955-7a4c-4710-8cf2-f610aa9bcd8f/registry/0.log" Jul 29 07:23:22.057849 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:22.057809 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-566654d898-kc6wh_12733955-7a4c-4710-8cf2-f610aa9bcd8f/registry/1.log" Jul 29 07:23:22.100857 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:22.100828 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hn27_1efa9f84-b540-41ee-b657-cbe318eff42c/node-ca/0.log" Jul 29 07:23:23.199703 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:23.199669 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cn74bt_88979fb6-2d23-4cf3-ae48-c41f48e23238/istio-proxy/0.log" Jul 29 07:23:23.454921 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:23.454833 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/0.log" Jul 29 07:23:23.505681 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:23.505651 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-kq7nr" Jul 29 07:23:23.512779 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:23.512755 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/1.log" Jul 29 07:23:24.596101 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:24.596078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs5xp_df1d2617-2492-4ea0-b258-c2a27e18d731/serve-healthcheck-canary/0.log" Jul 29 07:23:25.284647 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:25.284612 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72skd_bc9deaae-8679-4bd7-905f-d3457077a28b/kube-rbac-proxy/0.log" Jul 29 07:23:25.321724 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:25.321644 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72skd_bc9deaae-8679-4bd7-905f-d3457077a28b/exporter/0.log" Jul 29 07:23:25.360838 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:25.360806 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-72skd_bc9deaae-8679-4bd7-905f-d3457077a28b/extractor/0.log" Jul 29 07:23:27.046888 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:27.046851 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cn74bt_88979fb6-2d23-4cf3-ae48-c41f48e23238/istio-proxy/0.log" Jul 29 07:23:27.250262 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:27.250228 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/0.log" Jul 29 07:23:27.309542 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:27.309456 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/1.log" Jul 29 07:23:28.580774 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:28.580690 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-fvrsb_05fcf4a5-b597-448f-b6a8-7163b05fdc75/manager/0.log" Jul 29 07:23:28.612004 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:28.611978 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-fvrsb_05fcf4a5-b597-448f-b6a8-7163b05fdc75/copy-manifests/0.log" Jul 29 07:23:28.762580 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:28.762533 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cn74bt_88979fb6-2d23-4cf3-ae48-c41f48e23238/istio-proxy/0.log" Jul 29 07:23:28.996100 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:28.996069 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/0.log" Jul 29 07:23:29.051684 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:29.051638 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-w7h2m_1709b7a7-2b66-4056-909e-a24b27480c18/istio-proxy/1.log" Jul 29 07:23:30.175661 ip-10-0-131-31 kubenswrapper[2556]: I0729 07:23:30.175625 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68c9b6dddc-4vjgm_4d4d2a47-6657-4fb3-8b20-f05d87c29b59/manager/0.log"